No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Troubleshooting Guide

CloudEngine 16800, 12800, 12800E, 8800, 7800, 6800, and 5800 Series Switches

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Troubleshooting Procedure

Troubleshooting Procedure

If a slow service access fault occurs on a specified interface, perform the following steps to locate the fault (save operation records):
  1. Check whether the CPU usage of the device is high.

    If the value of System CPU Using Percentage in the display cpu [ slot slot-id ] command output is high (for example, higher than 70%), or the SYSTEM_1.3.6.1.4.1.2011.5.25.129.2.4.1 hwCPUUtilizationRisingAlarm alarm is generated, the CPU usage of the device is high. A high CPU usage often causes slow service access faults. For the common causes and troubleshooting procedure of a high CPU usage fault, see Guidance to Locating a High CPU Usage.

    If the CPU usage of the device is not high, go to step 2.

  2. Check whether congestion occurs on the intermediate link. (Perform this step in V100R003C10 and later versions.)

    Run the display qos buffer overrun history interface interface-type interface-number slot slot-id command in the diagnostic view to check historical records about interface traffic exceeding the buffer percentage threshold. (CE8800, CE7800, CE6800, and CE5800 switches do not support slot parameter.)
    <HUAWEI> system-view
    [~HUAWEI] diagnose
    [~HUAWEI-diagnose] display qos buffer overrun history interface 10ge 1/0/1
    -------------------------------------------------------------------
                       Overrun Time
    -------------------------------------------------------------------
          0                3834/778                 2014-08-30 18:31:06
          0                3834/778                 2014-08-30 18:33:06
          0                3834/778                 2014-08-30 18:35:06
          0                3834/778                 2014-08-30 18:37:06
          0                3834/778                 2014-08-30 18:39:06
          1                3738/759                 2014-08-30 18:31:06
          1                3807/773                 2014-08-30 18:33:06
          1                3795/770                 2014-08-30 18:35:06
          1                3675/746                 2014-08-30 18:37:06
          1                3665/744                 2014-08-30 18:39:06
    -------------------------------------------------------------------
    If no information is displayed, congestion does not occur on the interface. If the preceding command output is displayed, traffic rate on the interface exceeds the limit and congestion occurs. The QueueId field displays the ID of the queue where congestion occurs. The BufferUsage(Cell/KBytes) field displays the size of the buffer occupied by the congested queue.

    This command is used to check whether a few outgoing packets are discarded on the device and whether there is burst traffic on the device when the interface bandwidth usage is not high. To check the interface bandwidth usage, run the display interface brief | include up command in any view.

    • If congestion occurs, check whether rate limiting is configured. The procedure is as follows:
      • Check the rate limiting configuration in the inbound direction of an interface.
        <HUAWEI> display current-configuration | section include qos car inbound
        #                                                                               
                                                                  
        #                                                                               
        interface 10GE1/0/3                                                             
                

        The preceding command output shows that rate limiting in the inbound direction is configured on 10GE1/0/3 and the rate limit is 1 Gbit/s.

      • Check the rate limiting configuration in the outbound direction of an interface.
        <HUAWEI> display current-configuration | section include lr
        #                                                                               
        interface 10GE1/0/10                                                            
                                                           
         qos car inbound 222            

        The preceding command output shows that rate limiting in the outbound direction is configured on 10GE1/0/10 and the rate limit is 5000 kbit/s.

      • If rate limiting is configured and the traffic rate reaches the limit, cancel the rate limiting configuration or increase the rate limit. If congestion does not occur and service access is normal, the fault is rectified. Otherwise, go to step 3.
      • If rate limiting is not configured, consider network optimization or capacity expansion. If congestion does not occur after network optimization or capacity expansion, the fault is rectified. Otherwise, go to step 3.
      • If the CE12800, CE16800 or CE12800E is used, contact technical support personnel to check whether the bandwidth of the link between the LPU and SFU is sufficient. If the bandwidth is insufficient, add SFUs or replace the installed SFUs with higher performance SFUs. Otherwise, go to step 3.

        To remove an SFU, hold down the OFL button for 6s. You can remove the SFU until the OFL indicator is steady red.

    • If congestion does not occur, go to step 3.

  3. Check whether the device is attacked or MAC address flapping occurs.

    Perform the following steps:
    1. Run the display current-configuration | section include auto-defend command in the system view to check whether attack source tracing is enabled. By default, attack source tracing is disabled. If auto-defend enable is displayed in the command output, attack source tracing is enabled; otherwise, attack source tracing is disabled.
      • If attack source tracing is disabled, perform the following steps to enable the function, and check attack source information after a while.
        <HUAWEI> system-view
        [~HUAWEI] cpu-defend policy test
        [*HUAWEI-cpu-defend-policy-test] auto-defend enable
        [*HUAWEI-cpu-defend-policy-test] commit
      • If attack source tracing is enabled, run the display auto-defend attack-source command in any view to check attack source information.

        <HUAWEI> display auto-defend attack-source
          Attack Source User Table on Slot 4 :                            
          -------------------------------------------------------------------------                                                         
                Interface       PacketType    VLAN:Outer/Inner      Total                                                               
          -------------------------------------------------------------------------                                                         
          0000-c102-0102   10GE4/0/8       ICMP          1000/                 4832                
          -------------------------------------------------------------------------                                                         
          Total: 1                         
          Attack Source IP Table on Slot 4 :                                      
          -------------------------------------------------------------------------                                                         
                PacketType    Total                                                               
          -------------------------------------------------------------------------                                                         
                  ICMP          1144                                                                
          -------------------------------------------------------------------------                                                         
          Total: 1                         
        ...
      The preceding command output shows that the device discards 1144 ICMP packets from the attack source with the IP address 10.1.1.2.
      • If an attack source exists, locate the attack source and solve the problem. If service access is normal after the attack problem is solved, the fault is rectified.
      • If there is no attack source, or the fault persists after the attack problem is solved, go to step b.
    2. Run the display mac-address flapping [ slot slot-id ] [ begin YYYY/MM/DD HH:MM:SS ] command in any view to check whether MAC address flapping occurs.

      Earlier versions of V200R003C00:
      <HUAWEI> display mac-address flapping
      ...
      -------------------------------------------------------------------------------
      S  : start time    E  : end time    (D) : error down
      -------------------------------------------------------------------------------
      Time                  VLAN     Original-Port  Move-Ports     MoveNum
                            /BD                                                      
      -------------------------------------------------------------------------------
      S:2011-12-11 11:00:08 3     10GE1/0/1      10GE1/0/2      120  
      E:2011-12-11 11:33:13 /-
      
      
      -------------------------------------------------------------------------------
      Total items on slot 1: 1
      V200R003C00 and later versions:
      <HUAWEI> display mac-address flapping
      ...
      -------------------------------------------------------------------------------
      S  : start time    E  : end time    (D) : error down
      -------------------------------------------------------------------------------
      Time         : S:2017-08-24 14:40:11           E:2017-08-24 14:40:23      
      VLAN/BD      : 3/-          
        :               
      Original-Port: 10GE1/0/1         
      Move-Ports   : 10GE1/0/2     
      MoveNum      : 120 
      -------------------------------------------------------------------------------
      Total items on slot 1: 1
      If the preceding command output is displayed, MAC address flapping occurs.
      • If MAC address flapping occurs, locate the cause of MAC address flapping, and solve the problem. If service access is normal after the MAC address flapping problem is solved, the fault is rectified.

      • If MAC address flapping does not occur, or the fault persists after the MAC address flapping problem is solved, go to step 4.

  4. Check whether a device is added to the blacklist. If a device is incorrectly added to the blacklist, packets sent by the device will be discarded, resulting in the slow service access fault.

    Run the display cpu-defend blacklist statistics [ slot slot-id ] command in any view to check statistics on the packets sent to the CPU based on the blacklist. (The CE12800, CE16800 or CE12800E supports this command in V100R005C10 and later versions. The CE8800, CE7800, CE6800, and CE5800 do not support this command.)
    <HUAWEI> display cpu-defend blacklist statistics slot 3
    -------------------------------------------------------------------------------
    CPU defend policy b blacklist 1
    Slot 3
    -------------------------------------------------------------------------------
                         67, Droped Bytes                       6834
    -------------------------------------------------------------------------------
    The preceding command output shows that a device with the IP address 10.1.1.2 is on the blacklist, and 67 packets from the device are discarded. Perform the following steps to cancel the blacklist configuration:
    <HUAWEI> system-view
    [~HUAWEI] display current-configuration | section include blacklist     
    #
    cpu-defend policy b
     
     auto-defend enable
     auto-defend trace-type source-mac source-ip
     auto-defend protocol all
    [~HUAWEI] display acl 2001                              
    Basic ACL 2001, 1 rule
    ACL's step is 5
     
    [~HUAWEI] acl 2001                           
    [~HUAWEI-acl4-basic-2001] undo rule deny source 10.1.1.1 0
    [*HUAWEI-acl4-basic-2001] commit
    • If service access is normal after the blacklist configuration is canceled, the fault is rectified.
    • If the fault persists, go to step 5.
  5. Use the port mirroring function to obtain packets and check whether TCP packets are retransmitted.

    1. Run the observe-port [ observe-port-index ] interface interface-type interface-number command in the system view to configure a local observing interface.

    2. Run the port-mirroring observe-port observe-port-index { both | inbound | outbound } command in the interface view to mirror traffic to be monitored to the local observing interface. Analyze packets obtained on the interface to check whether TCP packets are retransmitted.

    After obtaining packets, check whether values of the sequence number field in some service packets are the same. If so, TCP packets are retransmitted, otherwise, TCP packets are not retransmitted.
    • If TCP packets are retransmitted, check whether the configuration of the local device causes TCP retransmission. TCP retransmission is often caused by packet loss or a long delay.
      • For details about how to check whether packet loss occurs, see Guidance to Locating Packet Loss.
      • If the delay is long, check whether the intermediate link is congested. Go to step 6.
    • If TCP packets are not retransmitted, go to step 6.
  6. Obtain packets using mirroring to check whether out-of-order delivery occurs. (For details about how to configure a local observing interface, see step 5.)

    Out-of-order delivery is often caused by the per-packet load balancing hash algorithm of an Eth-Trunk. The per-flow load balancing hash algorithm is recommended. To determine whether out-of-order delivery occurs, analyze the timestamp and sequence number field of service packets after obtaining packets. The value of sequence number in packets which arrive earlier should be less than that in packets which arrive later. If not, out-of-order delivery occurs.
    • If out-of-order delivery occurs, use the following method to check whether the device uses per-packet load balancing.

      <HUAWEI> display current-configuration | include hash
       eth-trunk hash-mode 4      

      If hash-mode is 4, the device uses per-packet load balancing. If hash-mode is not 4, the device uses per-flow load balancing.

      • If the device uses per-packet load balancing, change per-packet load balancing to per-flow load balancing.
      • If the device uses per-flow load balancing, check the upstream device using the preceding method.

      If the out-of-order delivery problem is solved, the fault is rectified. Otherwise, go to step 7.

    • If out-of-order delivery does not occur, go to step 7.
  7. Contact technical support personnel and collect diagnostic information displayed in the outputs of the following commands.

    • In V100R003C00 and V100R003C10 versions, run the display configuration diagnostic-information 9 process 3 or display configuration diagnostic-information 9 process 6 command in the diagnostic view.
    • In V100R005C00 and later versions, run the display configuration diagnostic-information luascript iffy process 3 or display configuration diagnostic-information luascript iffy process 6 command in the diagnostic view.
Translation
Download
Updated: 2020-01-07

Document ID: EDOC1000060766

Views: 615015

Downloads: 2962

Average rating:
This Document Applies to these Products

Related Version

Related Documents

Share
Previous Next