The CPU usage of the LPU in slot 2 reached 99%, with 80% occupied by the socket process. Ping to firewall interfaces failed, and all services were affected.
[USG9000-diagnose] display cpu-usage slot 2
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 99% Max: 99%
CPU Usage Stat. Time : 2014-05-10 09:32:56
CPU utilization for five seconds: 99%: one minute: 99%: five minutes: 99%.
TaskName CPU Runtime(CPU Tick High/Tick Low) Task Explanation
BOX 0% 0/ 0 BOX Output
DelPtTask 0% 0/ 0
CrtPtTask 0% 0/ 0
VCLK 0% 0/ 112a40
TRAF 0% 0/ 1c966 TRAFTraffic Statistics
CDM 0% 0/ 0 CDM
SOCK 80% 0/bf518c25 SOCKPacket schedule and processVTRU 0% 0/ 0 VTRUNK
Shut down GE2/1/9. The CPU usage of the socket process drops to a normal level.
1. Check the device status. No anomaly or alarm is detected.
2. Check session information. There are not too many sessions, and the CPU usage of SPUs is not high.
3. The socket process usage of the LPU in slot 2 is high, indicating that many packets are sent or processed by the LPU. These packets mainly include multicast protocol packets or ARP packets. You can run the display pe-counter 2 0 cause verbose clear command to view statistics about the cause. In the command, 2 indicates slot 2, and 0 indicates the LPUN (you can run the display version command to view the values).
[USG9000-diagnose] display pe-counter 2 0 cause verbose clear
Cause 0X08 = 1C19C3H(CAUSE_CPU_ARP)//Large number of ARP packets
Cause 0X13 = 28H(CAUSE_DROP_L2_DISABLE)
Cause 0X16 = 2H(CAUSE_CPU_ISIS)
Cause 0X33 = 1H(CAUSE_CPU_DIPV4_224)
Cause 0X44 = 13H(CAUSE_DROP_IPV4MC_DISABLE)
Cause 0X59 = A609H(CAUSE_DROP_AIB_FAKE )
Cause 0X5D = 11A095H(CAUSE_DROP_ECAR_RED)//A large number of packets are sent to the CPU and counted by CAR.
Cause 0X64 = 843H(CAUSE_CPU_ARP_MISS)
Cause 0X70 = AH(CAUSE_DROP_CIB_INVALID)
The output shows that the LPU receives many ARP packets, and many packets sent to the CPU are discarded due to CAR.
4. Run the display cpu-defend car protocol arp statistics slot 2 command to view the statistics on ARP packets discarded by the LPU.
[USG9000-diagnose] display cpu-defend car protocol arp statistics slot 2
Slot : 2
Application switch : Open
Default Action : Min-to-cp
IPV4 ARP packet
Protocol switch: N/A
Passed packet(s) : 19158490
Dropped packet(s): 985382873 //More than 900 million packets were discarded.
Configged CIR : 2000 kbps Actual CIR in NP : 2000 kbps
Configged CBS : 20000 bytes Actual CBS in NP : 20000 bytes
Priority : The index on this board can not be shown . Please see the NP Priority.
Min-packet-length : NA
The output shows that over 900 million ARP packets are discarded, which is abnormal. The ARP packets occupy 2000 kbit/s, interrupting the normal ARP service. In this case, the following conditions may occur:
a. The firewall cannot learn ARP entries in a timely manner and discards received ARP packets due to bandwidth insufficiency.
b. The firewall is busy sending ARP replies, leading to high CPU usage of the socket process.
Based on preceding analysis, we can primarily determine that the problem was caused by an ARP attack.
5. Check the packet sending and receiving status of the interfaces on the firewall. An anomaly is found on GE2/1/9.
[USG9000-diagnose]dis inter g2/1/9
Unicast: 809723711211 packets, Multicast: 14651208 packets
Broadcast: 1463277516 packets, JumboOctets: 352 packets
[USG9000-diagnose]dis inter g2/1/9
Unicast: 809723711211 packets, Multicast: 14651216 packets
Broadcast: 1466161798 packets, JumboOctets: 352 packets
The preceding command output shows that GE2/1/9 receives more than 3 million broadcast packets within 8 seconds, which are much more than the normal quantity, indicating that an ARP attack occurred on the interface. The interface connects to the Internet, indicating that the attack source is an upstream device.