S9703.CPU利用率过高

发布时间:  2015-01-09 浏览次数:  955 下载次数:  0
问题描述
故障现像:

S9703.单板利用率过高。

[IDC9703-1-diagnose]dis cpu-usage
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage            : 12% Max: 90%
CPU Usage Stat. Time : 2015-01-01  12:27:45
CPU utilization for five seconds: 12%: one minute: 11%: five minutes: 11%
Max CPU Usage Stat. Time : 2014-07-01 13:18:22.


[IDC9703-1-diagnose]dis cpu-usage  slot  3                
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage            : 84% Max: 89%
CPU Usage Stat. Time : 2015-01-01  12:27:25
CPU utilization for five seconds: 84%: one minute: 84%: five minutes: 84%
Max CPU Usage Stat. Time : 2014-12-24 05:44:54.


[IDC9703-1-diagnose]dis cpu-usage  slot  2
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage            : 87% Max: 94%
CPU Usage Stat. Time : 2015-01-01  12:27:19
CPU utilization for five seconds: 87%: one minute: 87%: five minutes: 87%
Max CPU Usage Stat. Time : 2015-01-01 10:26:29.

[IDC9703-1-diagnose]dis cpu-usage  slot  1        
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage            : 39% Max: 68%
CPU Usage Stat. Time : 2015-01-01  12:27:31
CPU utilization for five seconds: 39%: one minute: 39%: five minutes: 39%
Max CPU Usage Stat. Time : 2014-12-15 18:55:05.

告警信息
Jan  1 2015 07:05:27 IDC9703-1 %%01VOSCPU/4/CPU_USAGE_HIGH(l)[52]:Slot=3;The CPU is overloaded(CpuUsage=85%, Threshold=80%), and the tasks with top three CPU occupancy are:
bcmL2MOD.0  total      : 51%
PPI   total      : 14%
bcmCNTR.0  total      : 5%
Jan  1 2015 06:08:25 IDC9703-1 %%01INFO/4/SUPPRESS_LOG(l)[66]:Last message repeated 1 times.(InfoID=1079644206, ModuleName=VTY, InfoAlias=ACL_DENY)
Jan  1 2015 06:02:09 IDC9703-1 %%01INFO/4/SUPPRESS_LOG(l)[67]:Last message repeated 4 times.(InfoID=1079644206, ModuleName=VTY, InfoAlias=ACL_DENY)
Jan  1 2015 06:00:49 IDC9703-1 %%01SNMP/4/SNMP_FAIL(l)[68]:Failed to login through SNMP. (Ip=93.180.5.26, Times=2, Reason=the community was incorrect)
Jan  1 2015 05:54:16 IDC9703-1 %%01VOSCPU/4/CPU_USAGE_HIGH(l)[69]:Slot=3;The CPU is overloaded(CpuUsage=84%, Threshold=80%), and the tasks with top three CPU occupancy are:
bcmL2MOD.0  total      : 51%
PPI   total      : 13%
bcmCNTR.0  total      : 5%
#Dec 31 2014 23:57:34 IDC9703-1 L2IFPPI/4/MFLPVLANALARM:OID 1.3.6.1.4.1.2011.5.25.160.3.7 MAC move detected, VlanId = 88, MacAddress = d4ae-52a7-7e7d, Original-Port = GE3/0/31, Flapping port = GE3/0/32. Please check the network accessed to flapping port.
#Dec 31 2014 23:29:15 IDC9703-1 ARP/4/ARP_IPCONFLICT_TRAP:OID 1.3.6.1.4.1.2011.5.25.123.2.6 ARP detects IP conflict. (IP address=118.122.88.155, Local interface=GigabitEthernet3/0/44, Local MAC=782b-cb14-c9c5, Local vlan=88, Local CE vlan=0, Receive interface=GigabitEthernet3/0/45, Receive MAC=001b-219f-9f04, Receive vlan=88, Receive CE vlan=0, IP conflict type=Remote IP conflict).
#Dec 31 2014 23:29:15 IDC9703-1 ARP/4/ARP_IPCONFLICT_TRAP:OID 1.3.6.1.4.1.2011.5.25.123.2.6 ARP detects IP conflict. (IP address=118.122.88.147, Local interface=GigabitEthernet3/0/25, Local MAC=842b-2b63-dc61, Local vlan=88, Local CE vlan=0, Receive interface=GigabitEthernet3/0/26, Receive MAC=842b-2b63-dc60, Receive vlan=88, Receive CE vlan=0, IP conflict type=Remote IP conflict).
#Dec 31 2014 23:07:34 IDC9703-1 L2IFPPI/4/MFLPVLANALARM:OID 1.3.6.1.4.1.2011.5.25.160.3.7 MAC move detected, VlanId = 88, MacAddress = d4ae-52a7-7e7d, Original-Port = GE3/0/31, Flapping port = GE3/0/32. Please check the network accessed to flapping port.
处理过程
1.查看S9703单板CPU占用过高的进程。发现MSYN  进程异常。
2.查看LOG信息,发现3个问题:
   (1)2号单析和3号单板CPU利用率过高。
   (2)VLAN88的IP地址冲突。
   (3)GE3/0/31、GE3/0/32 有MAC地址漂移。
3.跟客户确认业务受影响的范围、确认VLAN88 的业务、网络拓扑。
4.查看设备运行的STP情况。设置的配置信息。
5.关闭 GE3/0/32,CP占用正常故障解除。
根因
环路导致单板CPU 利用过高

END