Some Services Are Interrupted on an NE40-X8 Running V600R003C00SPC300

The core routers of a headquarters are NE40E-X8s, and the branch routers are NE20E-X6s. Inter-AS Option A MPLS VPN is established between the headquarters and branches. The MPLS BGP peer relationship is set up properly, and routes can be learned correctly. However, VPN service gateways cannot ping each other.

Headquarters S5700-----Headquarters NE40E-X8----------Carrier's transmission device OSN-----------Branch NE20E-X6-------Branch S5700

Basic information:

Local device: NE40E-X8 V600R008C10SPC300; peer device: NE20E-X6 V600R008C10SPC300

Local device's lookback1 address: x.x.254.179; peer device's Lookback1 address: x.x.252.7

Local device's vpn_data gateway address: x.x.0.1/16; peer device's vpn_data gateway address: x.x.112.1/20

With vpn_data added, the local and remote devices can ping their gateways, and the MPLS status and BGP status are normal.

Symptom:

The local and remote devices with the source IP address specified cannot ping the gateway of the peer device.

1. Check the device configuration. It is found that the inter-AS Option A configurations are correct.

2. Check the route learning result on devices at both ends. It is found that the devices at both ends can properly learn the valid IP address of the peer gateway.

3. Perform the ping operation on both the local and remote devices to collect deice information and analyze the obtained packet header information.

Fast ping of large packets on the NE40E-X8:

<NE40-X8>ping -vpn-instance vpn_data -a  x.x.0.1 -s 1500 -c 1000000  -t  50  -m  50  x.x.112.1

Fast ping of large packets on the NE20E-X6:

<NE20-X6>ping -vpn-instance vpn_data -a  x.x.112.1 -s 1500 –c  1000000  -t  50 -m 50  x.x.0.1

Statistics on the involved interface:

Ping large packets on the NE40E, and the associated public network interface is G 4/0/0.

[NE40-X8-diagnose]display  interface GigabitEthernet  4/0/0

............

    Last 300 seconds input rate: 1136 bits/sec, 1 packets/sec——Traffic rate of the inbound interface during fast ping

    Last 300 seconds output rate: 65760 bits/sec, 19 packets/sec——Traffic rate of the outbound interface during fast ping

    Input: 72012304 bytes, 815065 packets

    Output: 381647128 bytes, 4332136 packets

    Input:

      Unicast: 92173 packets, Multicast: 561681 packets

      Broadcast: 161211 packets, JumboOctets: 0 packets——The count of received jumbo frames is always 0.

      CRC: 0 packets, Symbol: 0 packets  

      Overrun: 0 packets, InRangeLength: 0 packets

      LongPacket: 0 packets, Jabber: 0 packets, Alignment: 0 packets

      Fragment: 0 packets, Undersized Frame: 0 packets

      RxPause: 0 packets

    Output:

      Unicast: 194984 packets, Multicast: 3976194 packets

      Broadcast: 160958 packets, JumboOctets: 2206 packets——The number of jumbo frames received in the outbound direction during fast ping of large packets keeps increasing.

      Lost: 0 packets, Overflow: 0 packets, Underrun: 0 packets

      System: 0 packets, Overrun: 0 packets

      TxPause: 0 packets

      Unknown Vlan: 0 packets

    Input bandwidth utilization  :    0%

Output bandwidth utilization : 0.01%

The fast ping result on the NE40Es show that no jumbo count packets are received in response to a sent packet in the inbound direction and the rate of receiving packets in the inbound direction of an interface is far less than the rate of sending packets in the outbound direction. It can be basically confirmed that no packet is received in response to the ping packet.

Interface statistics about the peer NE20E-X6:

[NE20-X6-diagnose]display  interface GigabitEthernet  3/0/0

..........

    Last 300 seconds input rate: 1664 bits/sec, 2 packets/sec——The packet rate in the inbound direction is extremely small.

    Last 300 seconds output rate: 4912 bits/sec, 6 packets/sec

    Input: 70834013 bytes, 811627 packets

    Output: 56073719 bytes, 656225 packets

    Input:

      Unicast: 96669 packets, Multicast: 553661 packets

      Broadcast: 161297 packets, JumboOctets: 0 packets——The count of received jumbo frames is 0.

      CRC: 0 packets, Symbol: 0 packets

      Overrun: 0 packets, InRangeLength: 0 packets

      LongPacket: 0 packets, Jabber: 0 packets, Alignment: 0 packets

      Fragment: 0 packets, Undersized Frame: 0 packets

      RxPause: 0 packets

    Output:

      Unicast: 94027 packets, Multicast: 562196 packets

      Broadcast: 2 packets, JumboOctets: 0 packets

      Lost: 0 packets, Overflow: 0 packets, Underrun: 0 packets

      System: 0 packets, Overrun: 0 packets

      TxPause: 0 packets

      Unknown Vlan: 0 packets

    Input bandwidth utilization  :    0%

    Output bandwidth utilization : 0.01%

Statistics about packet count increasing during fast ping on the NE20E-X6 are as follows:

[NE20-X6-diagnose]display  interface GigabitEthernet  3/0/0

...........

   Last 300 seconds input rate: 4168 bits/sec, 7 packets/sec

    Last 300 seconds output rate: 67760 bits/sec, 27 packets/sec——The packet rate in the outbound direction is far greater than that in the inbound direction.

    Input: 71128583 bytes, 815686 packets

    Output: 59188594 bytes, 667547 packets

    Input:

      Unicast: 99352 packets, Multicast: 554734 packets

      Broadcast: 161600 packets, JumboOctets: 0 packets——The count of received jumbo frames is 0.

      CRC: 0 packets, Symbol: 0 packets  

      Overrun: 0 packets, InRangeLength: 0 packets

      LongPacket: 0 packets, Jabber: 0 packets, Alignment: 0 packets

      Fragment: 0 packets, Undersized Frame: 0 packets

      RxPause: 0 packets

    Output:

      Unicast: 104298 packets, Multicast: 563247 packets

      Broadcast: 2 packets, JumboOctets: 1529 packets——The number of jumbo frames in the inbound direction during ping of large packets keeps increasing.

      Lost: 0 packets, Overflow: 0 packets, Underrun: 0 packets

      System: 0 packets, Overrun: 0 packets

      TxPause: 0 packets

      Unknown Vlan: 0 packets

    Input bandwidth utilization  : 0.01%

    Output bandwidth utilization : 0.07%

Interface statistics on the peer NE40E:

<NE40-X8>display  interface GigabitEthernet  4/0/0

............

    Last 300 seconds input rate: 10168 bits/sec, 14 packets/sec

    Last 300 seconds output rate: 9072 bits/sec, 14 packets/sec

    Input: 73078191 bytes, 827191 packets

    Output: 389139633 bytes, 4356890 packets

    Input:

      Unicast: 102073 packets, Multicast: 563410 packets

      Broadcast: 161708 packets, JumboOctets: 0 packets——The number of received jumbo frames is 0.

      CRC: 0 packets, Symbol: 0 packets

      Overrun: 0 packets, InRangeLength: 0 packets

      LongPacket: 0 packets, Jabber: 0 packets, Alignment: 0 packets

      Fragment: 0 packets, Undersized Frame: 0 packets

      RxPause: 0 packets

    Output:

      Unicast: 206643 packets, Multicast: 3988791 packets

      Broadcast: 161456 packets, JumboOctets: 6002 packets

      Lost: 0 packets, Overflow: 0 packets, Underrun: 0 packets

      System: 0 packets, Overrun: 0 packets

      TxPause: 0 packets

      Unknown Vlan: 0 packets

    Input bandwidth utilization  :    0%

    Output bandwidth utilization :    0%

According to the fast ping result, the ping packets have been sent regardless of the fast ping of large packets on the NE40E or the ping of large packets on the NE20E-X6. However, the peer device has not received packets. Check the number of discarded packets on the forwarding chip. It is found that no packet loss count matches the packet ping rate. It is doubted that the fault may be on the transport devices on the intermediate network.

The data packets sent from the local device are discarded in the intermediate link consisting a carrier's devices.

After the carrier replace OSN550, services can be forwarded properly.