S9306环路故障导致全网终端大量丢包

发布时间:  2015-10-22 浏览次数:  1972 下载次数:  13
问题描述

版本:V200R007C00SPC500

组网拓扑:


组网概述:县局核心是S9306交换机,下挂楼层交换机和各个分支机构交换机,所有PC终端网关都在S9306上。

故障现象:全网PC终端上网速度慢,PC到网关之间大量丢包


告警信息

Oct 20 2015 15:28:07 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[34]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 15:26:44 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[35]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/38, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:24:15 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[36]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 15:24:09 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[37]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/3, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:23:33 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[38]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/6, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:13:35 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[39]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 15:12:10 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[40]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/20, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:11:43 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[41]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet1/0/2, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:11:19 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[42]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 15:10:54 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[43]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/38, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:08:31 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[44]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/3, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:06:15 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[45]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 15:03:51 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[46]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/20, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:02:21 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[47]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/4, AttackProtocol=ARP-REQUEST)

Oct 20 2015 15:01:09 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[48]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/1, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:58:15 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[49]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 14:55:13 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[50]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/38, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:53:35 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[51]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 14:52:59 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[52]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/3, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:48:29 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[53]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/20, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:47:00 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[54]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/4, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:40:33 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[55]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 14:39:36 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[56]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/38, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:39:36 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[57]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 14:35:14 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[58]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet1/0/2, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:29:05 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[59]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 14:28:41 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[60]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/20, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:26:03 GY_CXX_GA_S9306 %%01INFO/4/SUPPRESS_LOG(l)[61]:Last message repeated 1 times.(InfoID=4278652936, ModuleName=SECE, InfoAlias=PORT_ATTACK_OCCUR)

Oct 20 2015 14:23:56 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[62]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet3/0/38, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:23:34 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[63]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/2, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:20:59 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[64]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/4, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:17:37 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[65]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/20, AttackProtocol=ARP-REQUEST)

Oct 20 2015 14:16:03 GY_CXX_GA_S9306 %%01SECE/4/PORT_ATTACK_OCCUR(l)[66]:Auto port-defend started.(SourceAttackInterface=GigabitEthernet2/0/0, AttackProtocol=ARP-REQUEST)

处理过程

1、根据告警信息,查看有大量arp报文,查看设备运行状态正常。查看单板上丢弃大量arp-miss,arp-request报文

命令:display cpu-defend statistics slot 2

2、配置本机攻击防范和端口攻击防范,侦测攻击源192.168.0.234地址发送大量arp报文

命令:display auto-defend attack-source slot 2


3、配置MAC地址漂移检测,发现有MAC地址漂移

命令:display mac-address flapping recover 


4、配置环路检测,检测到环路动作为shutdown端口,定位到某个分支机构成环,断开该分支机构,网络恢复正常

5、到分支机构定位环路。发现该网络中使用SOHO交换机,设备默认地址为192.168.0.234,该设备会不停向全网发送免费ARP。

6、定位出环路线缆,解决网络故障



根因

1、网络中有环路

2、网络中的大量SOHO交换机,会不停发送免费arp,造成9306处理arp报文丢弃

解决方案

1、断开环路线缆

2、把所有的SOHO交换机地址取消

3、设置本机攻击防范

END