No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

The ARP limitation of upper network gateway S9300 caused network fault, leading the ONU run out of management

Publication Date:  2013-05-03 Views:  47 Downloads:  0
Issue Description
Topology: U2000----S9300 (V1R2C00SPC200) ----OLT----ONU
The gateway of U2000 and management IP of ONU is on S9300. The IP address of U2000 is 37.116.252.236.
More than 100 ONU run out of management from U2000 in a certain time. About 1 minute later the fault is disappeared.
Alarm Information
The communication between Network Management Tool and appliance that is managed is disconnected.


Handling Process
1. When the ONU is out of management, Ping the ONU from U2000. Ping test is failed. The disconnection of network is the reason.
2. Perform the traffic statistics on S9300, both the ICMP sent and received by U2000. The result shows that the S9300 has already forwarded the package sent from U2000 to ONU, but it does not receive the replay from ONU.
3. Perform the traffic statistics on Pon port that is connected between OLT and ONU. Apply the configuration as below. Note: the IP address of ONU that is out of management is 192.168.45.19.
 
Define ACL:
acl 3001
  rule 5 permit icmp source 37.116.252.236 0 destination 192.168.45.19 0
  acl 3002
  rule 5 permit icmp source 192.168.45.19 0 destination 37.116.252.236 0
 
Perform the traffic statistics on Pon port.
traffic-statistic outbound ip-group 3001 rule 5 port 0/2/0
  traffic-statistic inbound ip-group 3002 rule 5 port 0/2/0
 
Display the statistics information of OLT
MA5680T(config)#display qos-info all port 0/2/0
 
traffic-statistic:
  port 0/2/0:
  Inbound:
  Matches: Acl 3002 rule 5 running
  0 packet
  Outbound:
  Matches: Acl 3001 rule 5 running
  6 packets
 
From the result above it can be inferred that OLT has already forwarded the ICMP package sent from U2000 to ONU. But it has not received the reply from ONU. So the reason can be the ONU does not reply the Ping.
 
4. When the ONU is out of management, display the ARP on ONU.
MA5626(config)#display arp all
  { || }:
 
Command:
  display arp all
  IP Address MAC Address VLAN ID Port ONT Type
  192.168.45.3 781d-baaa-82b4 45 0/0 /1 - Dynamic
--- 1 entry found ---
From the result we can infer that the ONU does not have the ARP of the gateway. At this time ONU can’t ping the gateway successfully.
 
4. Enable the ARP debugging information of ONU
MA5626(diagnose)%%terminal debugging
  % Current terminal debugging is on
 
MA5626(diagnose)%%terminal monitor
  % Current terminal monitor is on
 
MA5626(diagnose)%%debugging arp packet
 
From the debugging information we can see that when the fault happens, ONU has sent the ARP request.
MA5626(diagnose)%%
  *0.861950990 MA5626 ARP/8/arp_send:Send an ARP Packet, operation : 1, sender_eth_addr : 286e-d4b1-0dc8,sender_ip_addr : 192.168.45.19,
  target_eth_addr : 0000-0000-0000, target_ip_addr : 192.168.45.1
  MA5626(diagnose)%%
  *0.861950990 MA5626 ARP/8/arp_send:Send an ARP Packet, operation : 1, sender_eth_addr : 286e-d4b1-0dc8,sender_ip_addr : 192.168.45.19,
  target_eth_addr : 0000-0000-0000, target_ip_addr : 192.168.45.1
  MA5626(diagnose)%%
  *0.861950990 MA5626 ARP/8/arp_send:Send an ARP Packet, operation : 1, sender_eth_addr : 286e-d4b1-0dc8,sender_ip_addr : 192.168.45.19,
  target_eth_addr : 0000-0000-0000, target_ip_addr : 192.168.45.1
 
But it does not received the response. Only after many requests has been sent does it receive the response.
 
MA5626(diagnose)%%
  *0.862041060 MA5626 ARP/8/arp_rcv:Receive an ARP Packet,
  operation: 2, sender_eth_addr: 0000-5e00-012d, sender_ip_addr: 192.168.45.1,
  target_eth_addr: 286e-d4b1-0dc8, target_ip_addr: 192.168.45.19
 
At this time, display the ARP, ONU has learned the ARP of gateway.
MA5626(diagnose)%%display arp all
  { || }:
 
Command:
  display arp all
  Total Scope(3) ARP Node Number : 3
  Total Waiting For Proxying Node Number : 0
  Total Dynamic ARP Node Number : 3
 
IP Address MAC Address VLANID Port ONT Status Age Type
  VPN
  192.168.45.3 781d-baaa-82b4 45 0/0 /1 - RESOLVED NO_AGE Dynamic
  192.168.45.1 0000-5e00-012d 45 0/0 /1 - RESOLVED NO_AGE Dynamic
  192.168.45.253 0000-0000-0000 45 65535/255 - RESOLVING INIT Dynamic
--- 3 entries found ---
Now, U2000 can ping ONU successfully. The out of management issue has disappeared.
 
5. From the information above, we can conclude that the ONU out of management is because of the disconnection between ONU and U2000. The disconnection is because the ARP of ONU is aged, but the ARP request sent to gateway has not been responded on time. The problem can be located on S9300. By default, S9300 has a limitation of rate for ARP request which is 64 Kbps. There are many services under S9300, the number ARP requests has exceeded the limitation which makes some of the ARP request can’t be responded on time.
Change the limitation of rate to 256 Kbps, the issue has been resolved.
 
Root Cause
The possible reason could be:
a. Network issue
b. S9300 issue.
C.U2000 issue
 

Suggestions
When debugging the out of management fault, first you will need to check the connectivity of network. In most cases this is the root cause. If the disconnection of network leads to the fault, you will have to capture the packets or perform the traffic statistics to debug and locate the root cause. The in-depth understanding of related protocols is a must to fix the issue.

Configuration Reference:
Below commands can be used to display statistics information of boards.
display cpu-defend arp-request configuration all
  display cpu-defend arp-request statistics all
  display cpu-defend arp-reply configuration all
  display cpu-defend arp-reply statistics all
 
 
#Define cpu-defend policy
  [Quidway]cpu-defend policy 1
  [Quidway-cpu-defend-policy-1]car packet-type arp-request cir 256
 
#Apply the policy to all interface boards.
[Quidway] cpu-defend-policy 1 global
 
 
#Apply the policy to a certain interface board.
  [Quidway]slot 3
  [Quidway-slot-3]cpu-defend-policy 1
 
#Apply the policy to main board.
[Quidway]cpu-defend-policy 1
 

END