Receiving Many STP TC Packets Causes a High CPU Usage on an S5700

Publication Date:  2015-06-09 Views:  1997 Downloads:  0
Issue Description
Figure 5-1 shows the networking of an enterprise. The CPU usage of an S5700 (S11) frequently exceeds 90%.

Figure 5-1 Receiving many STP TC packets causes a high CPU usage on an S5700


Alarm Information
Run the display cpu-usage command to check CPU information on the S5700. The CPU usage of the S5700 has reached 97%.

<S5700> display cpu-usage
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage            : 18% Max: 97%
CPU Usage Stat. Time : 2014-10-07  11:19:29
CPU utilization for five seconds: 18%: one minute: 18%: five minutes: 18%
Max CPU Usage Stat. Time : 2014-09-11 16:37:54.

View log information on the S5700. The log shows that a large number of TC packets have been processed.

Oct  7 2014 11:06:20-05:13 S5700 %%01INFO/4/SUPPRESS_LOG(l)[15]:Last message repeated 1 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct  7 2014 11:05:19-05:13 S5700 %%01INFO/4/SUPPRESS_LOG(l)[16]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct  7 2014 11:04:12-05:13 S5700 %%01INFO/4/SUPPRESS_LOG(l)[17]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Handling Process
Step 1 Related information was not collected when the fault occurred; therefore, it is unknown that which process triggers high CPU usage. There is a possibility that the FTS task triggers high CPU usage when processing many TC packets. The switch continuously generates TC packet logs. Check whether the TC packets are generated by the switch or received from other devices.

Run the display stp tc-bpdu statistics command on the S5700. The command output shows that the number of TC packets received by GigabitEthernet0/0/52, which is connected to SwitchA, keeps increasing, and the TC packets are forwarded to other access switches. Therefore, the TC packets are not generated by the S5700.

<S5700> display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port                        TC(Send/Receive)      TCN(Send/Receive)
0     GigabitEthernet0/0/51       29272/63              0/0
0     GigabitEthernet0/0/52       3/18363               0/0

Step 2 Run the display stp tc-bpdu statistics command on all the devices receiving the TC packets to find out the device where the TC packets are generated.

The command output shows that Eth-Trunk 1 on SwitchA has received a large number of TC packets. Eth-Trunk1 is connected to core SwitchB; therefore, the TC packets are not generated by SwitchA.

<SwitchA> display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port                        TC(Send/Receive)      TCN(Send/Receive)
0     GigabitEthernet0/0/1        16754/7               0/0
0     GigabitEthernet0/0/2        17112/1               0/0
0     GigabitEthernet0/0/3        17462/11              0/0
0     GigabitEthernet0/0/4        17793/4               0/0
0     GigabitEthernet0/0/5        18118/5               0/0
0     GigabitEthernet0/0/6        18415/3               0/0
0     GigabitEthernet0/0/14       17791/3               0/0
0     GigabitEthernet0/0/15       18113/6               0/0
0     GigabitEthernet0/0/16       18435/4               0/0
0     Eth-Trunk1                  4/11010               0/0

GigabitEthernet0/0/2 on SwitchB has received a large number of TC packets. GigabitEthernet0/0/2 is connected to GigabitEthernet0/0/52 of S4; therefore, the TC packets are not generated by SwitchB.

<SwitchB> display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port                        TC(Send/Receive)      TCN(Send/Receive)
0     GigabitEthernet0/0/1        12495/13               0/0
0     GigabitEthernet0/0/2        135/8349               0/0
0     GigabitEthernet0/0/3        13430/19               0/0
0     GigabitEthernet0/0/4        13784/14               0/0
0     GigabitEthernet0/0/5        14200/17               0/0
0     GigabitEthernet0/0/6        14687/10               0/0
0     GigabitEthernet0/0/14       14164/16               0/0
0     GigabitEthernet0/0/15       14164/16               0/0
0     GigabitEthernet0/0/16       14625/12               0/0
0     Eth-Trunk1                  11012/4               0/0

GigabitEthernet0/0/51 of S4 has sent a large number of TC packets.

<S4> display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port                        TC(Send/Receive)      TCN(Send/Receive)
0     GigabitEthernet0/0/51       8196/1123             0/0 
0     GigabitEthernet0/0/52       8343/136              0/0

Step 3 The command output on S4 shows that the number of outgoing TC packets keeps increasing; therefore, the TC packets are generated by S4. Run the display stp topology-change command on S4 to view TC packet information. The following information shows that the status of GigabitEthernet0/0/51 alternates between Blocked to Forwarding. When the port status changes to detected, topology changes.

<S4> display stp topology-change
CIST topology change information
   Number of topology changes             :8233
   Time since last topology change        :0 days 0h:0m:26s
   Topology change initiator(detected)    :GigabitEthernet0/0/51
   Number of generated topologychange traps :   9852
   Number of suppressed topologychange traps:   13

Step 4 Run the display interface brief command to view interface information on access devices. GigabitEthernet0/0/51 has received a large number of error packets. Wait for a period of time, and check the interface information again. GigabitEthernet0/0/51 still received a large number of error packets. This indicates that the fiber connected to this port is abnormal. After the fiber is replaced, the fault is rectified.

<S4> display interface brief
PHY: Physical
*down: administratively down
^down: standby
(l): loopback
(s): spoofing
(E): E-Trunk down
(b): BFD down
(e): ETHOAM down
(dl): DLDP down
(d): Dampening Suppressed
InUti/OutUti: input utility/output utility
Interface                   PHY   Protocol InUti OutUti   inErrors  outErrors   
........
GigabitEthernet0/0/51       up    up       0.01%  0.02%   38068638          0
........
----End
Root Cause
On an STP network, the quality of links between the devices participating in STP calculation degrades, causing frequent topology convergence and a large number of TC packets. Therefore, the CPU usage of the devices receiving the TC packets increase, affecting services.
Solution
Find out the reason why the quality of links between S3 and S4 degrades.
Suggestions
Run the stp tc-protection command globally on the core devices participating in STP calculation. After the command is executed, a device updates entries at most once every two seconds when the device frequently receives TC packets. This configuration prevents high CPU usage caused by frequent updates of MAC address entries and ARP entries.

END