No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade
Knowledge Base

High CPU usage on CE6800

Publication Date:  2019-07-10  |   Views:  1118  |   Downloads:  0  |   Author:  a84090745  |   Document ID:  EKB1001955996

Contents

Issue Description

Customer receves High CPU ussage errors 

 Oct 23 2018 10:19:28 ume1.dr1.a3.se %%01DEFEND/4/hwCpcarDropPacketAlarm_active(l):CID=0x807f042d-alarmID=0x09632007;Rate of packets to cpu exceeded the CPCAR limit in slot 1. (Protocol=TTL-EXPIRED, PPS/CBS=256/37000, ExceededPacketCount=8850)


Handling Process


Preliminary analysis shows the increased cpu usage could be caused by the high number of service traffic

 

===============================================================================
display cpu-defend statistics all
===============================================================================

Statistics(packets) on slot 1 :
--------------------------------------------------------------------------------
PacketType Total Passed Total Dropped Last Dropping Time
Last 5 Min Passed Last 5 Min Dropped
--------------------------------------------------------------------------------
arp-miss 228369369 1861098 2018-10-29 15:42
dhcp 3595473 1499 2018-10-28 10:55
fib-hit 9791310 52090 2018-10-26 02:59
telnet 801474 111122 2018-10-20 07:11
ttl-expired 19253743 299066 2018-10-29 15:42

 

l  arp-miss packets are generated when the device has a route to the destination IP address of a packet, but has no ARP entry matching the next hop of the route. This can be a normal situation when there is a large amount of traffic on the network

l  the large number of dhcp packets also suggest that the device is overloaded by the increased number of users

l  fib-hit indicates packets with the destination IP address being the local address; along with the high number of telnet packets, both indicate an increased network activity

l  the large number of ttl-expired packets suggest a possible routing loop in the network

We have asked customer to provide us inforamation about how many users were serviced before, and how much the user number has increased and advice him to check the routing table and peer devices for any Layer 3 loops


Optionally we asked him to configure auto-defend attack source tracing, in order to identify the source of TTL-expired packets:

#                                                                              

cpu-defend policy test1                                                                                                              

 car packet-type ttl-expired pps 128                                                   

 auto-defend enable                                                            

 auto-defend action deny                                                       

 auto-defend alarm enable                                                      

 auto-defend trace-type source-mac source-ip                   

 auto-defend protocol all                                                       

#  

cpu-defend-policy test1

He send us a graphs from device 




We have noticed that the  user number increased in just one day, therefore, an increase in the number of access users would explain the increased CPU usage.

Root Cause

Number of access users increased in just one day, causing high CPU alarm

Solution

This is a normal behavior.

Suggestions

Configure cpu-defend policy. 

The cpu-defend policy tracks the source of TTL-expired packets. This would allow to see the origin of packets that are being looped (source IP) as well as the peer router that is looping them back (source MAC)

After applying the policy, can be used the command as in the example below to trace the attack source.

 

<HUAWEI> display auto-defend attack-source

  Attack Source User Table on Slot 1 :                           

  -------------------------------------------------------------------------                                                        

  MAC Address      Interface       PacketType    VLAN:Outer/Inner      Total                                                              

  -------------------------------------------------------------------------                                                        

  0000-c102-0102   10GE1/0/1       ICMP          1000/                 4832               

  -------------------------------------------------------------------------                                                        

  Total: 1                        

  Attack Source IP Table on Slot 1 :                                     

  -------------------------------------------------------------------------                                                        

  IP Address      PacketType    Total                                                               

  -------------------------------------------------------------------------                                                        

  10.1.1.2        ICMP          1144