Analysis Report of Network Disconnection at a Site

Publication Date:  2013-07-31 Views:  241 Downloads:  0
Issue Description
After the Eudemon 1000E-U/X was deployed in place of the firewall of company A and was connected to the network, it was found that the global IP addresses of some NAT servers were inaccessible. One of these addresses automatically turned accessible 7 hours after it was connected to the network.

The MAC address of the Eth-Trunk port of the Eudemon 1000E-U/X was changed to the one of company A on January 19, and all addresses became accessible. At the noon of January 20, however, it was found that the network was disconnected, and users failed to ping the gateway from the firewall.

Alarm Information
None
Handling Process
1. The Eudemon 1000E-U/X was connected to the network on the nights of January 18 and 19, and it was found that the global IP addresses of NAT servers were inaccessible. A command was executed to have the Eudemon 1000E-U/X send gratuitous ARP messages, but the addresses remained inaccessible. The global IP addresses of NAT servers were configured on the port, but the addresses remained inaccessible.

2. At the noon of January 20, the network was suddenly disconnected. Users failed to ping the telecom gateway from the Eudemon 1000E-U/X, but could ping intranet addresses successfully. This means that the internal port of the firewall works normally while the external port fails.

3. At the night of January 21, the Eudemon 1000E-U/X was connected to the network, and packets were captured between the Eudemon 1000E-U/X and splitter B. The global addresses of the Eudemon 1000E-U/X were configured on the port and were pinged from a public network. Some addresses were inaccessible. The captured packets showed that, for all these inaccessible addresses, the destination MAC addresses of packets were not the address of the Eudemon 1000E-U/X. This means that the uplink device B of the Eudemon 1000E-U/X did not learn the ARP entries to these addresses, but used the original ARP entries. The ARP entries on device B is checked. It was found that 192.168.1.4, 192.168.1.10, and 192.168.1.13 corresponded to the MAC address of E0/1 of company A. Then we draw the conclusion: Some addresses were inaccessible after the Eudemon 1000E-U/X was connected to the network, because device B did not learn the ARP entries of these public addresses of the Eudemon 1000E-U/X.

4. At about 21:30, the network was suddenly disconnected. Users failed to ping 192.168.1.1 (telecom gateway address) from the Eudemon 1000E-U/X, but could ping 192.168.1.11 (address of splitter B) successfully. The situation was the same as that at the noon of January 20. At that moment, users could ping 192.168.1.1 and 192.168.1.11 successfully from the public network. ICMP request packets from the public network were captured between the gateway and splitter B, but ICMP reply packets were not captured. ICMP request packets from the public network were not captured between splitter B and the Eudemon 1000E-U/X. This means that packets were discarded by splitter B. When the telecom gateway was directly connected to the Eudemon 1000E-U/X, services recovered immediately. Then, the firewall of company A was connected to the network in place of the Eudemon 1000E-U/X. Services were disconnected when the firewall was connected to splitter B, but were connected when the firewall was connected to the telecom gateway. This situation is the same as that for the Eudemon 1000E-U/X. Therefore, the network disconnection was caused by splitter B, which failed to forward packets.
Users failed to ping 192.168.1.14 from the public network.Users failed to ping 192.168.1.1 from the firewall.

5. Finally, the original network topology was restored. Users could ping public network addresses from splitter B successfully. The ARP entries on splitter B were cleared and services recovered.
Root Cause
When splitter B already maintains ARP entries, it does not completely update its ARP entries according to the ARP packets from the Eudemon 1000E-U/X. As a result, some addresses on the Eudemon 1000E-U/X are inaccessible. To solve this problem, clear the ARP entries from splitter B to have it learn ARP entries again.
The network was disconnected after operating for a period because splitter B failed to forward packets.
In conclusion, splitter B has defect in ARP packet processing. It does not learn certain ARP entries, and causes the network to be disconnected after running for a period. Further location is required from experts on splitter B.
Suggestions
Directly connect the telecom gateway to the Eudemon 1000E-U/X, bypassing the splitter.

END