S93交换机互联口不配置undo port trunk allow-pass vlan 1导致ME60业务震荡问题

发布时间:  2014-12-19 浏览次数:  282 下载次数:  16
问题描述

【Problem Summary】调整用户侧链路后ME60脱管
【Problem Details】调整ME60用户侧链路后,ME60脱管,BGP协议中断,从直连设备ping不通ME60。

 

基本组网如下:

  NE5000E-1     ---------           NE5000E-2

  |                                   |

NE40-1          ---------            NE40-2

  |                                   |

3/0/0                               3/0/0

ME60-1  (3/0/1)  ---------  (3/0/1)  ME60-2

  |3/0/2                              |3/0/2

  |                                   |

  |_______________S93_________________|

 

 

ME60双机组网,现场做链路改造,将ME60之间的互联口由3/0/1口更换到6/0/0,将与交换机相连的下行口 3/0/2 用3/0/3来更换,对其他接口未做操作,在做完配置的时候,网管系统对ME602突然管理不上。

现场登录设备,在上行设备NE500E(在NE40E上面)上,查看ME60互联口的地址路由,学不到路由。

在NE40E上直接ping与ME602的直连地址无法ping通,持续时间大概24分钟后,恢复正常,一段时间后再次故障。同时查看ME60-2的BGP邻居,发现BGP邻居多次出现重新建。但是设备上的ISIS状态正常。

 

处理过程

1,根据反馈的信息,ISIS协议正常,BGP协议震荡。ISIS是基于二层协议的,BGP是基于TCP的,由此判断链路没有问题,是IP转发有问题。

2,登陆设备观察故障现象,确认故障时网络侧互连口的ARP建立失败,长时间处于Incomplete状态。

[BAS02]display  arp interface  GigabitEthernet  3/0/0
IP ADDRESS      MAC ADDRESS     EXPIRE(M) TYPE        INTERFACE   VPN-INSTANCE
                                          VLAN/CEVLAN PVC                     
------------------------------------------------------------------------------ 
10.1.1.1  xxxx-xxxx-d0e7            I -         GE3/0/0 
10.1.1.2  Incomplete      0         D-3         GE3/0/0
------------------------------------------------------------------------------
Total:2         Dynamic:1       Static:0     Interface:1   

3,查看单板的防攻击统计,确认存在大量的ARP请求上送,远远超过上送带宽,被大量丢弃,初步判断是ARP攻击导致正常ARP无法保持。

[BAS02-diagnose]display  cpu-defend statistics-all slot  3 clear
CarID Index Packet-Info                          Passed-Packets Dropped-Packets
--------------------------------------------------------------------------------
    6    15 EXCP_ID_IPV4_ARP                               5237        6351056 
  --------------------------------------------------------------------------------
[BAS02-diagnose]display  cpu-defend statistics-all slot  3 clear
CarID Index Packet-Info                          Passed-Packets Dropped-Packets
-------------------------------------------------------------------------------- 
    6    15 EXCP_ID_IPV4_ARP                               4388        5320211

 

4,查看攻击溯源信息,确认ARP攻击全部来自一个用户侧主接口3/0/3。

[BAS02-diagnose]display  attack-source-trace slot  3 verbose
Info: Please waiting.........     
Slot: 3
  Attack-source-trace Capacity:1M
  No.1 Packet Info:

  Interface Name    : GigabitEthernet3/0/3
  PeVlanid          : 0
  CeVlanid          : 0
  Attack Type       : CPCAR
  Attack Packet Time: 2014-12-02 17:10:09

  L2 Type: Ethernet
    Source Mac     : 00-00-xx-xx-xx-01
    Destination Mac: ff-ff-ff-ff-ff-ff
    Ethernet type  : 0x0806

  L2.5 Type: ARP
    Arp Type : Arp Request
    Source Ip: 0.0.0.0
    Dest Ip  : 255.255.255.255

  Attack Trace Data:     
     ff ff ff ff ff ff 00 00 5e 00 01 01 08 06 00 01 08 00 06 04 00 01 00 00 5e
     00 01 01 00 00 00 00 ff ff ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00
     00 00 00 00 00 00 00 00 00 00
  ----------------------------------
  No.2 Packet Info:
                                         
  Interface Name    : GigabitEthernet3/0/3
  PeVlanid          : 0
  CeVlanid          : 0
  Attack Type       : CPCAR
  Attack Packet Time: 2014-12-02 17:10:09

  L2 Type: Ethernet
    Source Mac     : 00-00-xx-xx-xx-01
    Destination Mac: ff-ff-ff-ff-ff-ff
    Ethernet type  : 0x0806

  L2.5 Type: ARP
    Arp Type : Arp Request
    Source Ip: 0.0.0.0
    Dest Ip  : 255.255.255.255

  Attack Trace Data:     
     ff ff ff ff ff ff 00 00 5e 00 01 01 08 06 00 01 08 00 06 04 00 01 00 00 5e
     00 01 01 00 00 00 00 ff ff ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00
     00 00 00 00 00 00 00 00 00 00
  ----------------------------------
  No.3 Packet Info:

  Interface Name    : GigabitEthernet3/0/3
  PeVlanid          : 0
  CeVlanid          : 0
  Attack Type       : CPCAR
  Attack Packet Time: 2014-12-02 17:10:09

  L2 Type: Ethernet
    Source Mac     : 00-00-xx-xx-xx-01
    Destination Mac: ff-ff-ff-ff-ff-ff
    Ethernet type  : 0x0806

  L2.5 Type: ARP
    Arp Type : Arp Request
    Source Ip: 0.0.0.0
    Dest Ip  : 255.255.255.255

  Attack Trace Data:     
     ff ff ff ff ff ff 00 00 5e 00 01 01 08 06 00 01 08 00 06 04 00 01 00 00 5e
     00 01 01 00 00 00 00 ff ff ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00
     00 00 00 00 00 00 00 00 00 00
  ----------------------------------

 

5,查看3/0/3接口的配置,确认主接口并没有部署用户业务,可以判定此流量是二层过来的非法流量。

 

[BAS02-GigabitEthernet3/0/3]dis th
#
interface GigabitEthernet3/0/3
undo shutdown

 

6,查看互连交换机的配置,如下配置情况下交换机是不应该发送不带VLAN的报文出来的。

 

[SW02-9312-GigabitEthernet1/0/4]dis this
#
interface GigabitEthernet1/0/4
description TO BAS02 G3/0/3
port link-type trunk
port trunk allow-pass vlan 90 to 91 2001 2007 2010 2019 2022 2025 2028 2514
undo negotiation auto
#
7,经过和交换机研发确认,在接口不配置 undo port trunk allow-pass vlan 1的情况下,交换机会误将其他所有VLAN的广播报文都往VLAN1上复制一份,最终从这个接口发出时就是不带VLAN的报文。

同时核对调整前接口的配置,确认调整前的接口上是有该配置的:

interface GigabitEthernet1/0/2
description TO BAS02- G3/0/2
port link-type trunk
undo port trunk allow-pass vlan 1
port trunk allow-pass vlan 90 to 91 2001 2007 2010 2019 2022 2025 2028 2514
undo negotiation auto

 

8,现场交换机上配置undo port trunk allow-pass vlan 1后,确认ME60侧ARP攻击记录解除,业务恢复。

[SW02-9312-GigabitEthernet1/0/4]      dis this                                             
#
interface GigabitEthernet1/0/4
description TO CQCK-PC-CMNET-BAS01-ME60-8 G3/0/3
port link-type trunk
undo port trunk allow-pass vlan 1
port trunk allow-pass vlan 90 to 91 2001 2007 2010 2019 2022 2025 2028 2514
traffic-policy 80disable inbound
undo negotiation auto

 

 

[BAS02-diagnose]display  cpu-defend statistics-all slot  3 clear
CarID Index Packet-Info                          Passed-Packets Dropped-Packets
-------------------------------------------------------------------------------- 

 

[BAS02]display  arp interface  GigabitEthernet  3/0/0
IP ADDRESS      MAC ADDRESS     EXPIRE(M) TYPE        INTERFACE   VPN-INSTANCE
                                          VLAN/CEVLAN PVC                     
------------------------------------------------------------------------------
10.1.1.1  xxxxx-xxxx-d0e7            I -         GE3/0/0 
10.1.1.2  xxxx-xxxx-cc04  20        D-3         GE3/0/0
------------------------------------------------------------------------------
Total:2         Dynamic:1       Static:0     Interface:1

 

根因
交换机互连口上未配置undo port trunk allow-pass vlan 1导致发出了大量的ARP攻击报文,攻击ME60,导致正常的ARP消息挤占,影响业务。
解决方案

问题原因:交换机vlan1 的配置未禁止导致ARP攻击ME60,引起正常ARP老化.

解决方案:交换机接口配置undo port trunk allow-pass vlan 1解决

建议与总结

涉及业务调整的,如果有与S93交换机互连的,务必关注交换机接口有无undo port trunk allow-pass vlan 1配置。

END