No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Boards Are Reset and Protocols Are Interrupted Because of Incorrect Configuration of Remote Port Mirroring

Publication Date:  2012-07-27 Views:  49 Downloads:  0
Issue Description

Network topology:
Network description:
The double devices of an ISP MAN are NE5KE routers of version 5.10 (CN) RELEASE 0044. Routes are learnt and advertised between the MAN egress and the national backbone network through EBGP. The NE5KE routers at the MAN egress forwards upstream traffic by advertising default OSPF routes. Interfaces G3/0/0 and G6/0/3 of HC_JNJ00_NE5K_1 are connected to the cyber café and G2/0/7 is connected to the packet analyzer.
Customer operations:
1. The customer configures port mirroring on HC_JNJ00_NE5K_ to mirror interfaces G3/0/0 and G6/0/3 to the two interfaces POS4/1/0 between the two NE5KE routers.
2. Then, the customer configures port mirroring on HC_JBJ00_NE5K_1 to mirror POS4/1/0 to the interface G2/0/7 connected a packet analyzer.
Fault description:
1. After the configuration, the OSPF neighbor relationship becomes Down 20 minutes later between the POS interfaces that connect HC_JNJ00_NE5K_1 and HC_JBJ00_NE5K_1. The OSPF neighbor relationship fails to be established again and the interface status is UNAVAILABLE.
2. The BGP neighbor relationship also becomes Down between the POS4/0/0 interfaces of HC_JBJ00_NE5K_1 and the device on the HC_JBJ00 national backbone network.
3. The CPU use rate is very high for a period.
4. The slot 4 of HC_JBJ00_NE5K_1 is restarted.
Doubts about the fault:
1. Can data stream be mirrored from GE interfaces to POS interfaces on HC_JNJ00_NE5K_1? If packets can be mirrored from GE interfaces to POS interfaces, are the packets outgoing from POS interfaces Ethernet frames or PPP packets?
2. How does HC_JBJ00_NE5K_1 process the mirrored packets that are received on POS 4/1/0?
3. Can the router HC_JBJ00_NE5K_1 mirror packets to the packet analyzer according to the configuration?
4. Why are the boards reset and protocols interrupted?
5. Will our subsequent products still support the remote mirroring function?

      
Alarm Information

1. Related logs on HC_JBJ00_NE5K_1:
%Sep 14 11:53:53 2007 HC_JBJ00_NE5K_1 SRM/4/PESError:PES error! LPU4:PEID 1:FPGAID 255:ExceptionID 0 (Lost heartbeat).
%Sep 14 11:53:39 2007 HC_JBJ00_NE5K_1 PES/4/Log_PEMS:Slot=4;
The PE1 has lost heartbeat.
%Sep 14 11:53:35 2007 HC_JBJ00_NE5K_1 LDP/5/LOG:

 Received TCP Up Event for TCP SockId 2
%Sep 14 11:53:32 2007 HC_JBJ00_NE5K_1 RM/4/RMLOG:OSPF 163 223: Nbr 218.14.111.197 event NegotiationDone State ExStart -> Exchange.
%Sep 14 11:53:25 2007 HC_JBJ00_NE5K_1 PES/4/Log_PEMS:Slot=4;
The PE1 has lost heartbeat.
%Sep 14 11:53:05 2007 HC_JBJ00_NE5K_1 LDP/5/LOG:

 Received TCP Up Event for TCP SockId 2
%Sep 14 11:53:04 2007 HC_JBJ00_NE5K_1 RM/4/RMLOG:OSPF 163 223: Nbr 218.14.111.197 event 2WayReceived State Init -> ExStart.
%Sep 14 11:53:04 2007 HC_JBJ00_NE5K_1 RM/4/RMLOG:OSPF 163 223: Nbr 218.14.111.197 event HelloReceived State Down -> Init.
%Sep 14 11:52:59 2007 HC_JBJ00_NE5K_1 RM/4/RMLOG:OSPF 163 223: Nbr 218.14.111.197 event InactivityTimer State Exchange -> Down.
[HC_JBJ00_NE5K_1-hidecmd]display lpureset 4 
LPU4 reset information:

-- 1. DATE:2007-09-14  TIME:11:53:53  RESET Num:1 
--    Reason:588 report error,and reset lpu!

2. Logs on HC_JNJ00_NE5K_1:
%Sep 14 12:23:42 2007 HC_JNJ00_NE5K_1 PPP/5/RejectOther:Slot=4;Pos4/1/0: Some protocol is rejected,it's not CHAP or PAP, PPP session will be closed.
%Sep 14 12:23:42 2007 HC_JNJ00_NE5K_1 PPP/5/RejectOther:Slot=4;Pos4/1/0: Some protocol is rejected,it's not CHAP or PAP, PPP session will be closed.
%Sep 14 12:23:42 2007 HC_JNJ00_NE5K_1 PPP/5/RejectOther:Slot=4;Pos4/1/0: Some protocol is rejected,it's not CHAP or PAP, PPP session will be closed.

            
Handling Process

1. After receiving the notification about the fault from customers, query and find out their operations and ask them to delete the port mirroring commands on devices at both ends.
2. Log on to the remote device and query the CPU information, registration of the board, logs, resetting information on the board, and status of protocols.
3. Lon on to the device remotely and observe the operation of the device for a period. Make sure that the device operates stably and the CPU of the device runs normally.
4. Tell customers about the principle of the port mirroring function. The observing port is forbidden to carry other services.  In addition, tell customers that the principle in devices of other vendors including Company C. In the document of Company C, the following are written:

When you configure a port as a SPAN destination port, the port is dedicated for use only by the SPAN feature. A SPAN destination port does not forward any traffic except that required for the SPAN session. 

            
Root Cause

The cause is that the customer doses not completely capture the principle of the port mirroring function and thus configures the function incorrectly. The port mirroring process is as follows:
1. On HC_JNJ00_NE5K_, data streams are mirrored from G3/0/0 and G6/0/3 to POS4/1/0. They pass through the SFU and are upstream forwarded to POS4/1/0. Then, they are sent out by POS4/1/0 with Ethernet frame headers.
2. The interface POS4/1/0 of HC_JBJ00_NE5K_1 receives the data streams, including data over POS links and mirrored Ethernet data sent from remote devices. The POS interfaces decapsulates the layer-2 data. Because POS interface can decapsulate only data frames over normal POS links, they process mirrored Ethernet frames by taking them as PPP packets. Therefore, the chip fails to identify the data frames. The upstream NP thus considers the data frames as unidentified PPP packets and sends them to the downstream NP through the SFU. The data frames are finally sent to CP.
3. Port mirroring is done on the upstream TM. Therefore, port mirroring is successful before the board of HC_JBJ00_NE5K_1 runs abnormally. That is, data packets are sent to the packet analyzer (mirroring server).
4. The downstream NP sends the PPP packets that are unable to be identified to the CP. The Cause value of such packets is 0A inside the router. The following hidden command can be used to display the CPCAR configuration of the Cause value of the corresponding slot:
The preceding information indicates that when the boards sends downstream PPP packets to the CP, the CIR = 1000 Kbps and the PIR = 0 Kbps (0 means that the concept of PIR is not supported by versions V2R1 and V2R2, or PIR = CAR).
5. Resetting of the board: The board continuously sends such packets to the CP to consume the CPU and a large number of abnormal packets may exist causing the upstream NP to work abnormally. Finally, heartbeat packets in the boards are lost and as a result, the board is reset.
6. Protocol interruption: The said situation and resetting of the boards both can interrupt protocols. Protocols are interrupted in the sequence of PPP, OSPF and then BGP.
7. Subsequent products also support the port mirroring function mirroring local data packets to the remote packet analyzer over MPLS tunnels.

                  
Suggestions

Get familiar with and capture the features and application scenario of the features of the devices so as to guide customers to deploy features and perform troubleshooting correctly.

      

END