S9306设备下挂用户发现有丢包

发布时间:  2017-01-11 浏览次数:  268 下载次数:  0
问题描述

某局点分公司一台9306设备,下挂用户发现有丢包现象。

 

1、查看设备CPU统计,可以看到设备上有大量arp-request报文丢弃

 

 

  <xxxxxxxS9306>display cpu-defend statistics packet-type arp-request all

 Statistics on mainboard:

-------------------------------------------------------------------------------

Packet Type         Pass(Bytes)  Drop(Bytes)   Pass(Packets)   Drop(Packets)

-------------------------------------------------------------------------------

arp-request          545453092k            0      8560292022               0

-------------------------------------------------------------------------------

 Statistics on slot 1:

-------------------------------------------------------------------------------

Packet Type         Pass(Bytes)  Drop(Bytes)   Pass(Packets)   Drop(Packets)

-------------------------------------------------------------------------------

arp-request          142906362k   907918986k      2111845265     13366617387

-------------------------------------------------------------------------------

 

<5S.HZ-lhc.HW9306>display cpu-defend statistics packet-type arp-request all

 Statistics on mainboard:

-------------------------------------------------------------------------------

Packet Type         Pass(Bytes)  Drop(Bytes)   Pass(Packets)   Drop(Packets)

-------------------------------------------------------------------------------

arp-request          545453147k            0      8560292880               0

-------------------------------------------------------------------------------

 Statistics on slot 1:

-------------------------------------------------------------------------------

Packet Type         Pass(Bytes)  Drop(Bytes)   Pass(Packets)   Drop(Packets)

-------------------------------------------------------------------------------

arp-request          142906422k   907919010k      2111846160     13366617745

-------------------------------------------------------------------------------

 

 

 

2、查看当前主控和接口板CPCAR值如下:

 

 

<xxxxxxxxS9306>display cpu-defend configuration packet-type arp-request all

Car Configurations on main board.

----------------------------------------------------------------------

Packet Name           Status   Cir(Kbps)   Cbs(Byte)  Queue  Port-Type

----------------------------------------------------------------------

arp-request          Enabled         128       24064      3         NA

----------------------------------------------------------------------

Car Configurations On Slot 1.

----------------------------------------------------------------------

Packet Name           Status   Cir(Kbps)   Cbs(Byte)  Queue  Port-Type

----------------------------------------------------------------------

arp-request          Enabled         64       12032      3        UNI

----------------------------------------------------------------------

<5S.HZ-lhc.HW9306>

 

 

3、设备当前ARP表项已超过5K

 

 

<xxxxxxS9306>display arp statistics all

                               Dynamic: 5280 Static: 7

 

4、分析造成单板CPU丢包的可能原因有两个:(1)有大量ARP攻击上送CPU,超过单板处理能力;(2)设备ARP表项规格超过5K,默认的CPCAR值满足不了当前业务需求;

 

 

5、设备上记录大量ARP报文攻击的日志:

 

 

Dec 27 2016 15:49:06 5S.HZ-lhc.HW9306 %%01SECE/4/USER_ATTACK(l)[467]:User attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/1, OuterVlan/InnerVlan=30/0, UserMacAddress=08c0-2104-d727, AttackPackets=15 packets per second)

Dec 27 2016 15:49:04 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[468]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/22, OuterVlan/InnerVlan=1900/0, AttackPackets=15 packets per second)

Dec 27 2016 15:48:56 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[470]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/18, OuterVlan/InnerVlan=4047/0, AttackPackets=15 packets per second)

Dec 27 2016 15:48:48 5S.HZ-lhc.HW9306 %%01SECE/4/USER_ATTACK(l)[474]:User attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/41, OuterVlan/InnerVlan=30/0, UserMacAddress=08c0-2104-d727, AttackPackets=15 packets per second)

Dec 27 2016 15:48:33 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[476]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/37, OuterVlan/InnerVlan=30/0, AttackPackets=15 packets per second)

Dec 27 2016 15:48:33 5S.HZ-lhc.HW9306 %%01SECE/4/USER_ATTACK(l)[477]:User attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/37, OuterVlan/InnerVlan=30/0, UserMacAddress=08c0-2104-d727, AttackPackets=15 packets per second)

Dec 27 2016 15:48:29 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[478]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/17, OuterVlan/InnerVlan=1294/1029, AttackPackets=15 packets per second)

Dec 27 2016 15:48:04 5S.HZ-lhc.HW9306 %%01SECE/4/SPECIFY_SIP_ATTACK(l)[484]:The specified source IP address initiates an attack.(Slot=MPU, SourceAttackIP = 210.87.129.211, AttackPackets=40 packets per second)

Dec 27 2016 15:48:03 5S.HZ-lhc.HW9306 %%01SECE/4/SPECIFY_SIP_ATTACK(l)[486]:The specified source IP address initiates an attack.(Slot=LPU1, SourceAttackIP = 10.208.2.45, AttackPackets=15 packets per second)

Dec 27 2016 15:48:03 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[487]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/24, OuterVlan/InnerVlan=3/0, AttackPackets=15 packets per second)

Dec 27 2016 15:48:03 5S.HZ-lhc.HW9306 %%01SECE/4/USER_ATTACK(l)[488]:User attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/24, OuterVlan/InnerVlan=3/0, UserMacAddress=e468-a3aa-1854, AttackPackets=15 packets per second)

Dec 27 2016 15:47:58 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[489]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/38, OuterVlan/InnerVlan=1987/0, AttackPackets=30 packets per second)

Dec 27 2016 15:47:43 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[491]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/33, OuterVlan/InnerVlan=4043/0, AttackPackets=15 packets per second)

Dec 27 2016 15:47:43 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[492]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/10, OuterVlan/InnerVlan=4047/0, AttackPackets=15 packets per second)

Dec 27 2016 15:47:42 5S.HZ-lhc.HW9306 %%01SECE/4/SPECIFY_SIP_ATTACK(l)[493]:The specified source IP address initiates an attack.(Slot=MPU, SourceAttackIP = 192.168.0.28, AttackPackets=85 packets per second)

Dec 27 2016 15:47:42 5S.HZ-lhc.HW9306 %%01SECE/4/USER_ATTACK(l)[494]:User attack occurred.(Slot=MPU, SourceAttackInterface=GigabitEthernet1/0/34, OuterVlan/InnerVlan=4043/0, UserMacAddress=3425-5d03-101c, AttackPackets=85 packets per second)

Dec 27 2016 15:47:39 5S.HZ-lhc.HW9306 %%01SECE/4/SPECIFY_SIP_ATTACK(l)[495]:The specified source IP address initiates an attack.(Slot=LPU1, SourceAttackIP = 10.208.3.2, AttackPackets=15 packets per second)

Dec 27 2016 15:47:39 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[496]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/5, OuterVlan/InnerVlan=30/0, AttackPackets=15 packets per second)

Dec 27 2016 15:47:39 5S.HZ-lhc.HW9306 %%01SECE/4/USER_ATTACK(l)[497]:User attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/5, OuterVlan/InnerVlan=30/0, UserMacAddress=0819-a627-1226, AttackPackets=15 packets per second)

Dec 27 2016 15:47:39 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[498]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/38, OuterVlan/InnerVlan=1994/0, AttackPackets=15 packets per second)

Dec 27 2016 15:47:13 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[501]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/38, OuterVlan/InnerVlan=1971/0, AttackPackets=15 packets per second)

Dec 27 2016 15:47:03 5S.HZ-lhc.HW9306 %%01SECE/4/PORT_ATTACK(l)[504]:Port attack occurred.(Slot=LPU1, SourceAttackInterface=GigabitEthernet1/0/43, OuterVlan/InnerVlan=4044/0, AttackPackets=15 packets per second)

 

 

 

处理过程
 

1、  配置攻击溯源,确认是否存在ARP攻击,配置命令如下:

    #

     cpu-defend policy main-board

 auto-defend enable

 auto-defend attack-packet sample 5

 auto-defend threshold 30

 undo auto-defend trace-type source-portvlan

 undo auto-defend protocol tcp igmp telnet ttl-expired

#

cpu-defend policy io-board

 auto-defend enable

 auto-defend attack-packet sample 5

 auto-defend threshold 30

 undo auto-defend trace-type source-portvlan

 undo auto-defend protocol tcp igmp telnet ttl-expired

#

cpu-defend-policy main-board

cpu-defend-policy io-board global

#

 

配置完成后,通过命令display auto-defend attack-source查看具体的攻击源

 

<xxxxxxxS9306>display auto-defend attack-source

  Attack Source User Table (MPU):

  -------------------------------------------------------------------------

      MacAddress       InterfaceName      Vlan:Outer/Inner      TOTAL 

  -------------------------------------------------------------------------

  0023-8950-93c3   GigabitEthernet1/0/25        1622            21805  

  08c0-2104-d727   GigabitEthernet1/0/1         30              29160  

  -------------------------------------------------------------------------

  Total: 2

 

  Attack Source IP Table (MPU):

  -------------------------------------

   IPAddress        TOTAL Packets 

  -------------------------------------

  21x.xx.192.x     22210  

  192.168.1.8      220400 

  20x.xx.79.42     21805  

  -------------------------------------

  Total: 3

 

根据溯源结果排查对应的IP或者MAC的设备,确认发送大量ARP请求报文是否正常,如果不正常,可以通过黑洞mac或者ACL黑名单进行过滤。

 

2、当前设备ARP数量已达到5K,单板默认的CPCAR值(主控板128kbit/s,接口板64kbit/s)满足不了当前业务需求,建议放大CPCAR值;ARP总数超过5K,建议主控板接口板CPCAR值配置为主控板配置512Kbit/s,接口板配置256Kbit/s。注意:放大CPCAR值会是单板CPU使用率上升,建议合理配置:

 

配置命令如下:

 #

cpu-defend policy main-board

car packet-type arp-request cir 512 cbs 96256

 auto-defend enable

 auto-defend attack-packet sample 5

 auto-defend threshold 30

 undo auto-defend trace-type source-portvlan

 undo auto-defend protocol tcp igmp telnet ttl-expired

#

cpu-defend policy io-board

 car packet-type arp-request cir 256 cbs 48128

auto-defend enable

 auto-defend attack-packet sample 5

 auto-defend threshold 30

 undo auto-defend trace-type source-portvlan

 undo auto-defend protocol tcp igmp telnet ttl-expired

#

cpu-defend-policy main-board

cpu-defend-policy io-board global

根因
 

设备收到大量ARP请求,超过单板CPU处理能力,导致无法及时处理,影响ARP表项生成,导致ping丢包。

解决方案

当前设备ARP数量已达到5K,单板默认的CPCAR值(主控板128kbit/s,接口板64kbit/s)满足不了当前业务需求,建议放大CPCAR值;ARP总数超过5K,建议主控板接口板CPCAR值配置为主控板配置512Kbit/s,接口板配置256Kbit/s。注意:放大CPCAR值会是单板CPU使用率上升,建议合理配置:

 

cpu-defend policy main-board

car packet-type arp-request cir 512 cbs 96256

 auto-defend enable

 auto-defend attack-packet sample 5

 auto-defend threshold 30

 undo auto-defend trace-type source-portvlan

 undo auto-defend protocol tcp igmp telnet ttl-expired

#

cpu-defend policy io-board

 car packet-type arp-request cir 256 cbs 48128

auto-defend enable

 auto-defend attack-packet sample 5

 auto-defend threshold 30

 undo auto-defend trace-type source-portvlan

 undo auto-defend protocol tcp igmp telnet ttl-expired

#

cpu-defend-policy main-board

cpu-defend-policy io-board global

END