Number of Discarded Outbound Packets Keeps Increasing on S9300 Interfaces

Publication Date:  2015-12-18 Views:  1639 Downloads:  0
Issue Description

1. Networking: Two S9303 switches are connected by an Eth-Trunk link, and each switch is connected to some servers. Some service traffic needs to be forwarded between the switches.


2. Symptom: The network management system (NMS) reports packet loss alarms on multiple interfaces, but services on the network are not affected. When an operator logs in to the switches and checks packet statistics on these interfaces, the Discard counter value keeps increasing. The following is an example:

GigabitEthernet2/0/31 
Output:  19040059771 packets, 15564833213514 bytes
  Unicast:                19033747559,  Multicast:                     4349024
  Broadcast:                  1963188,  Jumbo:                               0
  Discard:                    8140109,  Total Error:                         0
 
  Collisions:                       0,  ExcessiveCollisions:                 0
  Late Collisions:                  0,  Deferreds:                           0
  Buffers Purged:                   0

Alarm Information


Handling Process

1. Check the status of an interface where packet loss has occurred.


GigabitEthernet2/0/31 current state : UP 
Line protocol current state : UP 
Description:HUAWEI, Quidway Series, GigabitEthernet2/0/31 Interface 
Switch Port, TPID : 8100(Hex), The Maximum Frame Length is 9216 
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 4cb1-6c7e-da30 
Last physical up time   : 2015-09-16 07:06:52 
Last physical down time : 2015-09-16 06:57:30 
Port Mode: COMMON COPPER 
Speed : 1000,  Loopback: NONE 
Duplex: FULL,  Negotiation: ENABLE 
Mdi   : AUTO 
Last 300 seconds input rate 112067448 bits/sec, 27120 packets/sec 
Last 300 seconds output rate 130123712 bits/sec, 27490 packets/sec 
Input peak rate 985727944 bits/sec, Record time: 2015-09-12 05:29:10     
Output peak rate 974035880 bits/sec, Record time: 2015-09-15 09:52:04    //Historical peak outbound traffic rate 
 
Input:  16553717087 packets, 9444065199935 bytes 
  Unicast:                16550336032,  Multicast:                     1938229 
  Broadcast:                  1442826,  Jumbo:                               0 
  Discard:                          0,  Total Error:                         0 
 
  CRC:                              0,  Giants:                              0 
  Jabbers:                          0,  Fragments:                           0 
  Runts:                            0,  DropEvents:                          0 
  Alignments:                       0,  Symbols:                             0 
  Ignoreds:                         0,  Frames:                              0 
 
Output:  19040059771 packets, 15564833213514 bytes 
  Unicast:                19033747559,  Multicast:                     4349024 
  Broadcast:                  1963188,  Jumbo:                               0 
  Discard:                    8140109,  Total Error:                         0 
 
  Collisions:                       0,  ExcessiveCollisions:                 0 
  Late Collisions:                  0,  Deferreds:                           0 
  Buffers Purged:                   0 
 
    Input bandwidth utilization threshold : 100.00% 
    Output bandwidth utilization threshold: 100.00% 
    Input bandwidth utilization  : 11.21%                                 
    Output bandwidth utilization : 13.01%                                //Current bandwidth usage of outbound traffic


The historical peak outbound traffic rate is high, but the current traffic rate is much lower than the interface bandwidth. In addition, statistics collected multiple times show that the Discard counter value keeps increasing.


2. According to RFC2863, outbound packets will be discarded on an interface if the interface is congested due to traffic bursts. The traffic rate displayed on an interface is an average rate in the last 300 seconds and does not indicate the peak rate. To determine whether the packet loss is caused by traffic bursts, configure mirroring to obtain packet information and check the traffic chart to analyze the trend of traffic rates.

The mirroring configuration commands are as follows:

observe-port 1 interface GigabitEthernet3/0/24

interface GigabitEthernet2/0/31

port-mirroring to observe-port 1 outbound


The preceding traffic chart shows that the traffic rate on the interface reaches the peak value within certain milliseconds. Therefore, packets are discarded on the interface due to congestion, which is a normal situation.


Root Cause

Interfaces are congested due to high burst traffic rates. Packets cannot be buffered on these interfaces and are therefore discarded.

Solution

1. Increase the interface bandwidth. For example, replace GE interfaces with 10GE interfaces.

2. Find the source of the burst traffic and ensure that the source sends traffic at an even rate.

3. Configure traffic shaping on the upstream device to reduce traffic bursts.

Suggestions

1. The Discard counter value of outbound packets will keep increasing when the burst traffic rate exceeds the interface bandwidth and buffer capacity. As defined in RFC2863, the Discard counter indicates the number of packets dropped because of insufficient buffer capacity.


RFC 2863                The Interfaces Group MIB               June 2000
ifOutDiscards OBJECT-TYPE
    SYNTAX      Counter32
    MAX-ACCESS  read-only
   STATUS      current
    DESCRIPTION
            "The number of outbound packets which were chosen to be
            discarded even though no errors had been detected to prevent
            their being transmitted.  One possible reason for discarding
            such a packet could be to free up buffer space.
 
            Discontinuities in the value of this counter can occur at
            re-initialization of the management system, and at other
            times as indicated by the value of
            ifCounterDiscontinuityTime."


2. Congestion may occur in the following conditions:

a. Some servers connected to a switch send burst traffic within a short period. For example, if the traffic rate is 1000 Mbit/s, the millisecond-level average traffic rate is 1 Mbit/ms. If traffic of more than 1 Mbits is transmitted within 1 ms, the peak rate will reach or exceed 1 Gbit/s. The burst traffic needs to be buffered on the line card to mitigate the traffic burst. If the traffic burst lasts for more than several milliseconds, the buffer will be full. As a result, packets are discarded.

b. Traffic is sent from a high-bandwidth interface to a low-bandwidth interface or from multiple interfaces to one interface.


3. See the following figure to display the traffic chart on Wireshark.



4. Congestion-triggered packet loss usually occurs on GE interfaces. In such case, use a 10GE interface to replace the congested GE interface.

END