What Is a Microburst? How to Detect a Microburst?

Monitoring Port Traffic Through the NMS

Why cannot devices support millisecond-level traffic statistics? This is because switches have a large number of ports. When a switch collects packet statistics on ports, the CPU traverses all ports, lowering the performance of obtaining port packet statistics. As a result, the CPU cannot traverse all ports to obtain traffic information during microbursts. In addition, millisecond-level polling on all ports greatly consumes CPU resources, which may affect normal services of the switch. Therefore, the switch cannot capture the microburst information.

Monitoring the Maximum Rate of Outgoing Packets on the Observing Port

When microbursts (packet loss) occur on the network, the maximum rate of outgoing packets on the observing port of a switch is far lower than the port rate, and the port usage is even lower than 10%.

The forwarding chip of the switch records only the total number of received packets. The rate of any packet on the switch can only be calculated by dividing the total number of packets in a period of time by the measurement period. Therefore, the measurement accuracy depends on the length of the measurement period. By default, Huawei CloudEngine series switches calculate the average values of the peak and instantaneous traffic rates based on a period of 300s. The measurement period used for calculating the peak rate is configurable, and the minimum value is 10s. That is, the statistics accuracy is 10s.

Using Telemetry to Monitor Microbursts

Telemetry is a device feature in a narrow sense while is a closed automatic O&M system in a board sense. It is divided into the device side and OSS side, and consists of network devices, collector, analyzer, and controller, which can be provided by Huawei or third parties. In Huawei Telemetry system, the network devices refer to CloudEngine switches, the collector and analyzer both refers to iMaster NCE-FabricInsight, and the controller refers to iMaster NCE-Fabric.

Telemetry can be used to detect microbursts on a network. Figure 1-7 shows the Telemetry system architecture and data processing flow.

Figure 1-7 Telemetry system architecture and data processing flow

The following details operations and data processing flow of Huawei Telemetry system.

1. Create a Telemetry static subscription on the device for microburst monitoring.
   1. Create a destination group to which a destination collector belongs, and configure the IP address and port number of the destination collector as well as the protocol and encryption mode for data sending.

      system-view
      telemetry        //Enter the Telemetry view.
      destination-group destination-group-name       //Create a destination group to which a destination collector belongs and enter the destination group view.
      ipv4-address ip-address port port [ vpn-instance vpn-instance ] [ protocol grpc [ no-tls ] ]      //Configure the IP address and port number of the destination collector as well as the protocol and encryption mode for data sending.
      commit

   2. Configure a sampling sensor group, including the sampling path and filter criteria. When the configured sampling path meets the filter criteria, the device reports the sampling path to the collector in a timely manner.

      The following describes how to configure the sampling path and filter criteria for a customized event.

      1. Create a sampling sensor group.

         system-view
         telemetry   //Enter the Telemetry view.
         sensor-group sensor-name     //Create a sampling sensor group and enter the sensor group view.

      2. Configure a sampling path for the customized event and enter the customized event view.
         CloudEngine switches only support the following sampling path for microburst monitoring: huawei-qos: qos/qosQueueBufUsageStats/qosQueueBufUsageStat. The minimum accuracy is 100 ms.

         sensor-path huawei-qos:qos/qosQueueBufUsageStats/qosQueueBufUsageStat self-defined-event

      3. Create a filter for the sampling path.

         filterfilter-name    //Create a filter for the sampling path and enter the filter view.
         op-field field op-type { eq | gt | ge | lt | le } op-value value     //Configure filter criteria.
         condition-relation { and | or }     //Configure the logical operation mode between filter criteria.
         commit

   3. Create a static subscription.

      The static subscription is used to associate the destination group where the destination collector is located with a sampling sensor group and configure gRPC information.

      1. Create a subscription.

         system-view
         telemetry     //Enter the Telemetry view.
         subscription subscription-name       //Create a subscription for associating the destination group where the destination collector is located with a sampling sensor group, and enter the subscription view.
         destination-group destination-name   //Associate the subscription with the destination group where the destination collector is located.
         sensor-group sensor-name   //Associate the subscription with a sampling sensor group.

      2. (Optional) Configure gRPC information.
         All the following configuration items have default values. If the default values meet the requirements, skip this step.

         protocol gRPC [ no-tls ]         //Configure the encryption mode for gRPC. By default, no encryption mode is configured for gRPC.
         local-source-address ipv4 ip-address     //Configure the source IP address for packets to be sent to the CPU. By default, the source IP address is the IP address of the route's outbound interface selected by the socket.
         dscp value     //Configure the DSCP value for data packets to be sent to the CPU. By default, the DSCP value is 0.
         encoding { json | gpb }    //Configure the encoding format for data packets to be sent to the CPU. By default, the encoding format is GPB.

      3. (Optional) Configure the maximum usage of the MPU's CPU resources for Telemetry data sampling.
         By default, Telemetry data sampling can occupy at most 5% of the MPU's CPU resources.

         cpu-usage max-percent usage

      4. Commit the configuration.

         commit

2. Configure sampling parameters for microburst monitoring on the device, including the upper and lower buffer thresholds and sampling interval.
   1. Configure the lower and upper buffer thresholds.

      system-view
      interface interface-type interface-number
      qos [ queue queue-index ] buffer-monitoring percent low low-percent high high-percent    //Configure the lower and upper buffer thresholds for a queue.
      quit     //Exit the interface view.

      When any of the following conditions is met, the device considers that a microburst occurs and then records microburst information, including the microburst occurrence time and corresponding queue buffer usage:

      • If the obtained queue buffer usage is higher than the upper buffer threshold or the lower buffer threshold is set to 0:
        • The difference between the obtained queue buffer usage and the last recorded queue buffer usage is greater than 2%.
        • The difference between the current microburst occurrence time and the last recorded microburst occurrence time is greater than 1 second.
      • If the obtained queue buffer usage is between the upper and lower buffer thresholds:
        • The difference between the obtained queue buffer usage and the last recorded queue buffer usage is greater than 2%, and a microburst occurred last time.
        • The difference between the current microburst occurrence time and the last recorded microburst occurrence time is greater than 1 second, and a microburst occurred last time.
   2. Configure the sampling interval.

      telemetry         //Enter the Telemetry view.
      subscription subscription-name       //Enter the view of the created static subscription.
      sensor-group sensor-name sample-interval sample-interval          //Configure the sampling interval for the associated sampling sensor group.
      commit

3. When the Telemetry sampling interval is reached, the device sends microburst information recorded during this period in Telemetry data format to the collector.
   Frequent reading the buffer usage of interface queues affects the CPU usage of a device. In this case, the device adjusts the Telemetry sampling interval based on the CPU usage.

   • When the CPU usage is lower than or equal to 60%, the device reads the buffer usage of interface queues at the configured sampling interval.
   • When the CPU usage is higher than or equal to 80%, the device stops reading the buffer usage of interface queues.
   • When the CPU usage is higher than 60% and lower than 80%, the device reads the buffer usage of interface queues based on the sampling ratio of 10:1. That is, the device reads the buffer usage of interface queues once every 10 sampling intervals.

4. The analyzer reads and then analyzes the data sent by the collector (Huawei iMaster NCE-FabricInsight).
   1. On the device side, configure the interconnection between iMaster NCE-FabricInsight and a device.
      1. Configure a route for the iMaster NCE-FabricInsight collector cluster to access the data center network.
      2. Configure SNMPv3 to add the device to iMaster NCE-FabricInsight for management.

         system-view 
         snmp-agent sys-info version v3 
         snmp-agent mib-view included iso-view iso    //Create a MIB view and name it iso-view. To ensure that iMaster NCE-FabricInsight can properly manage the device, the MIB view must contain the iso node.
         snmp-agent group v3 snmpv3group privacy write-view iso-view notify-view iso-view    //Create an SNMPv3 group, name it snmpv3group, and associate the write view and notify view named iso-view with the group. By default, the write view has the read permission. Therefore, you do not need to set the read view. The notify view is used to restrict the MIB objects of the device for sending traps to iMaster NCE-FabricInsight.
         snmp-agent usm-user v3 snmpv3user group snmpv3group      //Create an SNMPv3 user and name it snmpv3user, which must be consistent with the security name on iMaster NCE-FabricInsight. The user's security level cannot be lower than the security level of the user group to which the user belongs. Otherwise, the user cannot communicate normally. For example, if the security level of the user group snmpv3group is privacy, the security level of the user snmpv3user must be authentication and encryption.
         snmp-agent usm-user v3 snmpv3user authentication-mode sha2_512     //Set the authentication protocol and authentication password of the user, which must be the same as those on iMaster NCE-FabricInsight. The authentication protocol is SHA2_512 and the authentication password  is 8937561bc.
         Please configure the authentication password (8-255) 
         Enter Password: 
         Confirm Password:
         snmp-agent usm-user v3 snmpv3user privacy-mode aes256     //Set the encryption protocol and encryption password of the user, which must be the same as those on iMaster NCE-FabricInsight. The encryption protocol is AES256 and the encryption password is 68283asd.
         Please configure the privacy password (8-255) 
         Enter Password: 
         Confirm Password: 
         snmp-agent trap enable 
         snmp-agent target-host trap address udp-domain destip-address source loopback 0 udp-port 10162 params securityname snmpv3user v3 privacy     //destip-address is the IP address of iMaster NCE-FabricInsight. (If both the analyzer and collector are deployed, set destip-address to the floating IP address of the collector. If only the analyzer is deployed, set destip-address to the southbound floating IP address of the analyzer.) Loopback 0 is the interface where the source address of trap packets is located. 10162 is the port number of trap packets. You are not advised to change the port number. If you need to, contact technical support personnel. Set securityname to the user name.
         snmp-agent packet max-size 12000    //Set the maximum size of an SNMP packet that the SNMP agent can receive and send to 12000 bytes. The defaul value is 12000 bytes. Since this parameter may be modified, you need to reset this parameter.
         commit

      3. Configure the Syslog protocol so that the device reports exception logs to iMaster NCE-FabricInsight for analysis and display. The logs can be used for issue analysis and troubleshooting assistance.

         system-view 
         info-center enable     //Enable the information center.
         info-center channel 6 name fabricSyslog     //Configure the information center channels, set the channel number to 6, 7, or 8, and set the channel name to fabricSyslog.
         info-center loghost destip-address source-ip source-ip-address { public-net | vpn-instance vpn-instance-name } port 28305 channel fabricSyslog level debugging transport tcp ssl-policy policy1    //Configure the device to send logs of the debugging level to iMaster NCE-FabricInsight.
         undo info-center statistic-suppress enable //Disable suppression of statistics about consecutive repeated logs.
         commit

   2. On iMaster NCE-FabricInsight, perform the configuration for interconnecting with devices, that is, add devices to iMaster NCE-FabricInsight. The procedure is as follows:
      1. Configure an SNMPv3 template.
         1. Choose Inventory from the navigation tree and choose Protocol Template > SNMP.
         2. Click Create and set parameters in the template based on the SNMP protocol information. SNMP parameter settings on iMaster NCE-FabricInsight must be consistent with those on devices.
         3. Click Confirm to create the SNMP protocol parameter template.
      2. Add a single device to iMaster NCE-FabricInsight.
         1. Choose Inventory from the navigation tree and choose Network Resource > Device.
         2. Click Add Device.
         3. In the Basic Information area, set IP Address, Name, Fabric, and Device Role for the device.
         4. In the SNMP area, set SNMP parameters.
         5. Click Confirm. The device is added.
   3. On the following pages of iMaster NCE-FabricInsight, you can view queue metrics.
      1. Choose Telemetry from the navigation tree and click the Queue tab. Check the used buffer distribution of queues.
         Figure 1-8 Queue Used Buffer Distribution
         
      2. View queue details, including the queue buffer trend, packet loss trend due to queue congestion, and packet loss details due to queue congestion.
         Figure 1-9 Queue Details
         
   4. When a microburst occurs, iMaster NCE-FabricInsight identifies it as a health issue based on abnormal events.
      1. Choose Health from the navigation tree and choose the Issue tab.
      2. When a microburst occurs, you can check the Service Affected by Switch Port Congestion issue to view the trends of packet loss and packet loss details due to queue congestion, that is, the number of lost packets at different time points.

5. Based on the analysis result, the controller (Huawei iMaster NCE-Fabric) adjusts the network accordingly.
   1. On the device side, complete the configuration for interconnecting with iMaster NCE-Fabric.
      1. Configure a route for iMaster NCE-Fabric to access the data center network.
      2. Configure SNMPv3 to add devices to iMaster NCE-Fabric for management. For details, see 4.a.ii.
   2. On iMaster NCE-Fabric, perform the configuration for interconnecting with the device, that is, add the device to iMaster NCE-Fabric. The procedure is as follows:
      1. Access the Auto Discovery page.

         Choose Network Planning and Construction > Physical Resource > Device > Device Management. The Device Management page is displayed. Select Auto Discovery from the Device Discovery drop-down list box.

      2. Set SNMPv3 parameters.

         Select Customized and set SNMPv3 parameters.

      3. Click Start. The controller starts to discover devices.

         The discovered devices are displayed on the Discovered Devices tab page.

   3. Open the Loop Closure app and choose Event Management from the main menu.
   4. On the Fault Object tab page, view Service Affected by Switch Port Congestion and rectify the fault based on the suggestions provided on the page.
      Figure 1-10 Handling suggestions for Service Affected by Switch Port Congestion
      

Monitoring Microbursts Using Packet Capture and Traffic Analysis Tools

You can use packet capture and traffic analysis tools (such as Wireshark) to monitor microbursts on a network. The procedure is as follows:

1. Use the packet capture function to capture packets and generate a packet file.
   1. Configure outbound mirroring on the interface where packets are discarded.
   2. Mirror traffic to the observing port working at the same rate.
   3. On a PC or server, use the packet capture and traffic analysis tool to capture packets on the observing port and generate a packet file.

      Due to performance limitations, a PC can only capture packets at a maximum rate of 1 Gbps, and a server can capture packets at a maximum rate of 10 Gbps. You are advised to use a high-performance PC. Otherwise, the packet capture function may be affected.

2. Use the traffic analysis function to monitor microbursts.
   1. On the packet capture and traffic analysis tool, open the file of captured packets.
   2. You can view the traffic graph in the IO graph.
   3. Change the traffic unit for the Y axis in the IO graph to Bits and the interval to 1 ms. In this way, you can view millisecond-level traffic bursts.
      Figure 1-11 Wireshark IO graph
      

Monitoring Microbursts Using the Discarded Packets Capture Function

You can configure the discarded packet capture function to detect microbursts. The procedure is as follows:

1. Enable the discarded packet capture function so that the device captures outgoing packets on the ports where microbursts occur.

   system-view 
   qos capture drop-packet enable tcb egress

2. Check information about captured discarded packets.

   display qos capture statistics drop-packet { ip | ipv6 | ethernet } [ tcb ] slot slot-id

Traffic Analysis

1. Access layer
   1. North-south traffic: For example, when each access device has two uplink 40GE interfaces connected to the aggregation layer and 48 downlink 10GE interfaces connected to servers, the oversubscription ratio is 6:1, indicating a high risk of microbursts.
   2. East-west traffic: In the storage, computing, and Hadoop clusters, multiple servers may transmit traffic to one server at the same time, posting a high risk of microbursts.
2. Aggregation layer
   1. North-south traffic: North-to-south traffic has no risk of microbursts because the access traffic is greater than public network traffic. South-to-north traffic may pass through a firewall or not. South-to-north traffic passing through a firewall has a low oversubscription ratio and accounts for only 10% to 25% of the total traffic in the data center, indicating a low risk of microbursts. In addition, switches have much higher forwarding capabilities than firewalls, so the bottleneck lies in the firewalls. Similarly, the risk of microbursts is also low for south-to-north traffic that does not pass through a firewall.
   2. East-west traffic: Devices in the east and west directions have the same bandwidth, and the oversubscription ratio is 1:1. Therefore, the risk of microbursts is low.

Preventive Measures

Question

How do I determine whether a microburst occurs on an interface where packets are discarded and the bandwidth usage is low?

Products and Versions Involved

Products involved: CE6851HI, CE6855HI, CE6856HI, CE6860EI, and CE6865EI

Versions involved: V200R005C10 and later versions

Answer

1. Install the patch of V200R005C10SPH025 or a later version.
2. Log in to the device and enter the user view.
3. Enable high-precision microburst detection on the interface, and set a microburst detection duration.

   high-precision rate detection interface interface duration monitortime

   In this command, monitortime specifies the microburst detection duration. The value is an integer in the range from 1 to 1440, in minutes. Set this parameter based on the frequency at which the value of DISCARD is updated on an interface on the live network.

   An example is as follows:

   <HUAWEI>high-precision rate detection interface 10GE 2/0/20 duration 10
   Info: High-precision rate detection is enabled, duration 10 minutes.

4. If the value of DISCARD on the interface increases, check the peak rate of the interface to determine whether a microburst occurs.

   display high-precision rate detection interface interface

   An example is as follows:

   <HUAWEI>display high-precision rate detection result interface 10GE 2/0/12 
   Interface                     : 10GE2/0/12
   Configured detection duration : 10 Min
   Detection start time          : 2020-09-28 10:24:19
   Detection end time            : --
   Average input rate            : 0 Mbits/sec
   Average output rate           : 0 Mbits/sec
   
   Top 10 input rate:
   -----------------------------------------------
   Time                     Input  Rate(Mbits/sec)
   -----------------------------------------------
   2020-09-28 10:24:38.912                       0
   2020-09-28 10:24:38.915                       0
   2020-09-28 10:24:38.909                       0
   2020-09-28 10:24:38.903                       0
   2020-09-28 10:24:38.900                       0
   2020-09-28 10:24:38.893                       0
   2020-09-28 10:24:38.896                       0
   2020-09-28 10:24:38.884                       0
   2020-09-28 10:24:38.887                       0
   2020-09-28 10:24:38.881                       0
   -----------------------------------------------
   
   Top 10 output rate:
   -----------------------------------------------
   Time                     Output Rate(Mbits/sec) //Check the Output Rate column. In this example, the peak rate of the 10GE interface reached 10000 Mbit/s, indicating that a microburst occurred.
   -----------------------------------------------
   2020-09-28 10:24:29.345                    10000 
   2020-09-28 10:24:21.799                    10000
   2020-09-28 10:24:31.751                    10000
   2020-09-28 10:24:38.648                    10000
   2020-09-28 10:24:20.178                    10000
   2020-09-28 10:24:24.954                    10000
   2020-09-28 10:24:26.718                    10000
   2020-09-28 10:24:22.270                    10000
   2020-09-28 10:24:22.129                    10000
   2020-09-28 10:24:20.767                    10000
   -----------------------------------------------

5. Disable high-precision microburst detection on the interface.

   undo high-precision rate detection interface interface

   An example is as follows:

   <HUAWEI>undo high-precision rate detection interface 10GE 2/0/12
   Info: High-precision rate detection is disabled.