S7700交换机端口流量异常问题分析

发布时间:  2017-04-19 浏览次数:  627 下载次数:  0
问题描述

某客户反馈S7700交换机部分端口出方向流量达到100%,影响部分用户的付费点播业务。交换机版本为V200R003C00SPC500。

处理过程

1)定性判断异常端口的流量构成。在客户反馈的问题接口(G1/7/0/5)及故障VLAN199)上部署流量统计,发现出方向的流量中已知单播流量(目的MAC0024-6805-4039VLAN 199 单播流)远小于未知单播流量。

[BN_JH_ZXJF2F_S7712]display traffic policy statistics interface GigabitEthernet 1/7/0/5 outbound verbose rule-base

Interface: GigabitEthernet1/7/0/5

Traffic policy outbound: tj

Rule number: 5

Current status: OK!

Statistics interval: 300

---------------------------------------------------------------------

Classifier: tj operator or

Behavior: tj

Board : 1/7

rule 5 permit destination-mac 0024-6805-4039 vlan-id 199

---------------------------------------------------------------------

Passed | Packets: 343,493

| Bytes: 467,836,168

| Rate(pps): 0

| Rate(bps): 0

---------------------------------------------------------------------

Dropped | Packets: 0

| Bytes: 0

| Rate(pps): 0

| Rate(bps): 0

---------------------------------------------------------------------

rule 30 permit vlan-id 199

---------------------------------------------------------------------

Passed | Packets: 537,320

| Bytes: 731,817,611

| Rate(pps): 0

| Rate(bps): 0

---------------------------------------------------------------------

Dropped | Packets: 0

| Bytes: 0

| Rate(pps): 0

| Rate(bps): 0

---------------------------------------------------------------------

[BN_JH_ZXJF2F_S7712]

从出方向流量统计的结果来看,VLAN199的报文居多,且端口出方向流量以单播为主,说明出方向的流量大部分走的是未知单播。


2)抓包分析异常流量构成。通过客户抓包反馈的异常流量,反馈了3个异常流量的MAC地址,分别为:0024-6805-403b0024-6805-42300024-6805-424f。


3)检查MAC地址学习情况,发现这三条MAC地址设备没有学习到,由此可判断该端口出方向发出了目的地址不是该端口连接设备的异常单播流量,确认异常的流量属于未知单播引起的泛洪。

[BN_JH_ZXJF2F_S7712]display mac-address vlan  199 | in 4230

-------------------------------------------------------------------------------

MAC Address    VLAN/VSI                          Learned-From        Type     

-------------------------------------------------------------------------------

 

-------------------------------------------------------------------------------

[BN_JH_ZXJF2F_S7712]display mac-address vlan  199 | in 424f

-------------------------------------------------------------------------------

MAC Address    VLAN/VSI                          Learned-From        Type     

-------------------------------------------------------------------------------

 

[BN_JH_ZXJF2F_S7712]display mac-address vlan  199 | in 403b

-------------------------------------------------------------------------------

MAC Address    VLAN/VSI                          Learned-From        Type     

-------------------------------------------------------------------------------

 

-------------------------------------------------------------------------------

<BN_JH_ZXJF2F_S7712>display mac-address vlan  199

-------------------------------------------------------------------------------

MAC Address    VLAN/VSI                          Learned-From        Type     

-------------------------------------------------------------------------------

0008-e3ff-fd90 199/-                             Eth-Trunk17         dynamic  

0024-6803-85b2 199/-                             GE1/7/0/35          dynamic  

0024-6803-8844 199/-                             XGE1/11/0/6         dynamic  

0024-6803-a5b2 199/-                             GE1/7/0/36          dynamic  

0024-6803-a60f 199/-                             GE1/7/0/1           dynamic  

0024-6803-a844 199/-                             XGE1/11/0/6         dynamic  

0024-6805-4035 199/-                             GE1/7/0/2           dynamic  

0024-6805-4037 199/-                             GE1/7/0/7           dynamic  

0024-6805-4038 199/-                             GE1/7/0/0           dynamic  

0024-6805-4039 199/-                             GE1/7/0/5           dynamic  

0024-6805-403a 199/-                             XGE1/12/0/2         dynamic     

0024-6805-403c 199/-                             Eth-Trunk11         dynamic  

0024-6805-415f 199/-                             Eth-Trunk8          dynamic  

0024-6805-4160 199/-                             Eth-Trunk1          dynamic     

0024-6805-424d 199/-                             GE1/7/0/8           dynamic  

0024-6805-424e 199/-                             GE1/7/0/9           dynamic     

0024-6805-44a3 199/-                             XGE1/12/0/2         dynamic  

0024-6805-44a4 199/-                             XGE1/12/0/2         dynamic  

0024-6805-51e9 199/-                             XGE2/11/0/7         dynamic  

0024-6805-6035 199/-                             GE1/7/0/3           dynamic  

0024-6805-6036 199/-                             XGE2/11/0/9         dynamic  

0024-6805-6039 199/-                             GE1/7/0/6           dynamic  

0024-6805-603b 199/-                             XGE1/12/0/2         dynamic  

0024-6805-603c 199/-                             Eth-Trunk11         dynamic  

0024-6805-615f 199/-                             Eth-Trunk8          dynamic  

0024-6805-6160 199/-                             Eth-Trunk1          dynamic  

0024-6805-61e9 199/-                             XGE2/11/0/7         dynamic  

0024-6805-624d 199/-                             XGE2/11/0/10        dynamic  

0024-6805-624f 199/-                             GE1/7/0/45          dynamic  

b888-e377-7aab 199/-                             GE1/6/0/7           dynamic  

-------------------------------------------------------------------------------

Total items displayed = 30


4)通过客户提供异常流量的目的IP地址,交换机主动Ping这些地址后,查询ARP表项,可以看到相应的设备所在的端口。

<BN_JH_ZXJF2F_S7712>dis arp network 10.182.131.141

IP ADDRESS      MAC ADDRESS     EXPIRE(M) TYPE        INTERFACE   VPN-INSTANCE

                                          VLAN/CEVLAN

------------------------------------------------------------------------------

10.182.131.141  0024-6805-4230  20        D-0/0       GE1/7/0/11

                                           199/-

------------------------------------------------------------------------------

 

<BN_JH_ZXJF2F_S7712>dis arp | i 0024-6805-403b

IP ADDRESS      MAC ADDRESS     EXPIRE(M) TYPE        INTERFACE   VPN-INSTANCE

                                          VLAN/CEVLAN

------------------------------------------------------------------------------

10.182.131.146  0024-6805-403b  20        D-0/0       GE2/7/0/13

------------------------------------------------------------------------------

 

[BN_JH_ZXJF2F_S7712]display arp | in 0024-6805-424f

IP ADDRESS      MAC ADDRESS     EXPIRE(M) TYPE        INTERFACE   VPN-INSTANCE

                                          VLAN/CEVLAN

------------------------------------------------------------------------------

10.182.131.187  0024-6805-424f  20        D-0/0       GE1/7/0/44

------------------------------------------------------------------------------


5)通过绑定静态MAC地址表项的方式,将上述MAC绑定到相应的源端口上,业务流量恢复正常。

[BN_JH_ZXJF2F_S7712]mac-address static 0024-6805-424f GigabitEthernet 1/7/0/44 vlan 199

[BN_JH_ZXJF2F_S7712]mac-address static 0024-6805-403b GigabitEthernet 2/7/0/13 vlan 199

[BN_JH_ZXJF2F_S7712]mac-address static 0024-6805-4230 GigabitEthernet1/7/0/11 vlan 199

<BN_JH_ZXJF2F_S7712>display inter br | in up

PHY: Physical

*down: administratively down

^down: standby

(l): loopback

(s): spoofing

(E): E-Trunk down

(b): BFD down

(e): ETHOAM down

(dl): DLDP down

(d): Dampening Suppressed

InUti/OutUti: input utility/output utility

Interface                   PHY   Protocol InUti OutUti   inErrors  outErrors

GigabitEthernet1/7/0/0      up    up          0%     0%          0          0

GigabitEthernet1/7/0/1      up    up          0%    15%          0          0

GigabitEthernet1/7/0/2      up    up          0%    16%          0          0

GigabitEthernet1/7/0/3      up    up          0%    23%          0          0

GigabitEthernet1/7/0/5      up    up          0%    26%          0          0

GigabitEthernet1/7/0/6      up    up          0%     0%          0          0

GigabitEthernet1/7/0/7      up    up          0%  2.72%          0          0

GigabitEthernet1/7/0/8      up    up          0%     0%          0          0

GigabitEthernet1/7/0/9      up    up          0%  7.22%          0          0

GigabitEthernet1/7/0/10     up    up          0%     0%          0          0

GigabitEthernet1/7/0/11     up    up          0%    10%          0          0

GigabitEthernet1/7/0/23     up    up          0%    20%          0          0

GigabitEthernet1/7/0/25     up    up          0%     0%          0          0

GigabitEthernet1/7/0/26     up    up        0.71%  3.43%          0          0

GigabitEthernet1/7/0/27     up    up        0.81%  3.53%          0          0

GigabitEthernet1/7/0/29     up    up        1.55%  8.75%          0          0

GigabitEthernet1/7/0/30     up    up        4.12%    19%          0          0

GigabitEthernet1/7/0/31     up    up        4.30%    13%          0          0

GigabitEthernet1/7/0/32     up    up        2.51%    11%          0          0

GigabitEthernet1/7/0/34     up    up        0.44%  1.41%          0          0

GigabitEthernet1/7/0/35     up    up          0%    18%          0          0


根因

综上所述,导致业务流量上升的主要原因是原本走已知单播的流量,由于原端口MAC地址没有学习到,导致设备走了未知单播变为广播泛洪。

解决方案

问题的短期解决方案是将对业务敏感的VLAN 199下挂的服务器MAC地址,进行MAC地址静态绑定,防止大流量泛洪。

期来看,建议客户将问题单板升级成规格更高的单板,或者将问题单板上的部分业务迁移到其它单板上。

建议与总结

MAC地址转发时,采用相同的HASH算法去查找对应的VLAN+MAC表项,如果无法找到对应的表项,则产生流量泛洪。通过历史日志也能看到带宽超限曾经发生过多次,且在泛洪时存在MAC使用率超过阈值的告警,说明前期设备上发生的泛洪情况,与用户量增长,MAC使用率增加有关。

END