Adjusting the Buffer and PFC Thresholds
Generally, the VIQ function can meet service requirements in the network.
If the device does not support the VIQ and the service performance does not meet requirements or even packet loss occurs, you can manually adjust the buffer and PFC thresholds using commands.
Optimizing the Lossless Service Performance in Scenarios Where Both TCP and RoCE Traffic Is Transmitted
If both TCP and RoCE traffic needs to be transmitted, you are advised to manually reduce the shared buffer available to TCP traffic queues after configuring the VIQ function. This ensures more shared buffer space of the chip available to lossless RoCE traffic.
Recommended settings:
Table 2-9 lists the recommended dynamic threshold configurations of the shared buffer for a queue. Note that if the VIQ function has been enabled, the dynamic shared buffer threshold for lossless queues is automatically adjusted to the maximum value. To manually adjust the threshold for lossless queues, disable the VIQ function first.
<HUAWEI> system-view [~HUAWEI] interface 100ge 1/0/1 [*HUAWEI-100GE1/0/1] qos buffer queue 1 shared-threshold dynamic 1
Optimizing Lossless Service Performance in Scenarios Without Packet Loss
After automatic buffer optimization of lossless queues is enabled, if no packet loss occurs in lossless queues but service performance does not meet requirements (for example, the throughput is high but the delay is long), and the display dcb pfc interface command output shows that the number of received and sent PFC frames per second exceeds 5000, it can be determined that the PFC threshold is frequently triggered. In this case, you can increase the PFC threshold.
# Display statistics about the sent and received PFC frames.
<HUAWEI> display dcb pfc interface 100ge 1/0/1
-----------------------------------------------------------------------------------------
Interface Queue Received(Frames) ReceivedRate(pps) DeadlockNum
Transmitted(Frames) TransmittedRate(pps) RecoveryNum
-----------------------------------------------------------------------------------------
100GE1/0/1 3 148990 99 0
0 0 0
-----------------------------------------------------------------------------------------
In addition, after automatic buffer optimization of lossless queues is enabled, the switch allocates buffer resources to all panel ports. If multiple ports on the switch are not in use, you can increase the threshold for triggering PFC frames to increase the buffer threshold of the ports in use by occupying the buffer space of unused ports. This enhances the queue's capability to accommodate burst traffic, prevents PFC from being triggered frequently, and improves the performance of distributed RoCEv2 applications. When adjusting the PFC threshold, in addition to ensuring the performance of RoCEv2 services, do not set the PFC threshold to a too high value. Otherwise, backpressure cannot be performed in a timely manner upon network congestion, which may cause packet loss on egress queues.
To adjust the threshold for triggering PFC frames, change the value of the xoff parameter in the dcb pfc buffer command.
Recommended settings:
<HUAWEI> system-view [*HUAWEI] interface 100ge 1/0/1 [*HUAWEI-100GE1/0/1] dcb pfc buffer 3 xoff dynamic 4
Model |
Number of Uplink and Downlink Ports on a Single Device (N) |
Dynamic Threshold |
|---|---|---|
CE6865-48S8CQ-EI, CE8861-4C-EI |
≤ 8 |
6 (Remaining buffer coefficient: 1/2) |
[8,32] |
5 (Remaining buffer coefficient: 1/4) |
|
> 32 |
4 (Remaining buffer coefficient: 1/8) |
|
CE8850-64CQ-EI |
≤ 8 |
6 (Remaining buffer coefficient: 1/2) |
[8,16] |
5 (Remaining buffer coefficient: 1/4) |
|
> 16 |
4 (Remaining buffer coefficient: 1/8) |
|
For the CloudEngine 16800, the XOFF parameter supports only the static threshold. The recommended value is calculated according to the formula below. The value is in cells (for the CloudEngine 16800, one cell is 288 bytes). In the preceding formula, N indicates the maximum number of downlink ports on which congestion may occur, and M indicates the Incast level of traffic. For example, if the traffic model on the live network is 3:1, the value of M is 3. |
||
Precautions:
When an interface is transmitting traffic and encountering a congestion, the threshold for triggering PFC frames may fail to be configured on the interface. Therefore, you are advised to configure the threshold when the interface is not congested or after the interface is shut down using the shutdown command. (To check whether the interface is congested, run the display qos buffer-usage command.)
Resolving Packet Loss Issues
If no packet loss occurs on the live network, you can increase the PFC threshold to enhance the capability to accommodate burst traffic and optimize service performance. However, once packet loss occurs, you need to locate the cause and modify the configuration as soon as possible to prevent packet loss caused by improper buffer configurations. Packet loss on a chip includes upstream packet loss and downstream packet loss. Upstream packet loss occurs in the inbound direction on a switch port, and downstream packet loss occurs in the outbound direction on a switch port. The possible causes and handling measures of packet loss vary according to the packet loss location.
To check whether packet loss occurs, run the display interface command. If the value of Discard is not 0 in Input, upstream packet loss occurs. If the value of Discard is not 0 in Output, downstream packet loss occurs.
<HUAWEI> display interface 100GE1/0/1
100GE1/0/1 current state : UP (ifindex: 5)
Line protocol current state : UP
Description:
Switch Port, TPID : 8100(Hex), The Maximum Frame Length is 9216
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 1457-9f9e-2f71
Port Mode: COMMON FIBER, Port Split/Aggregate: DISABLE
Speed: 100000, Loopback: NONE
Duplex: FULL, Negotiation: DISABLE
Input Flow-control: DISABLE, Output Flow-control: DISABLE
Mdi: -, Fec: RS-FEC
Last physical up time : 2019-05-13 02:48:35
Last physical down time : 2019-05-13 02:45:08
Current system time: 2019-05-14 09:05:59
Statistics last cleared:2019-05-14 03:18:20
Last 10 seconds input rate: 8425269499 bits/sec, 976871 packets/sec
Last 10 seconds output rate: 57453998 bits/sec, 42560 packets/sec
Input peak rate 12349559550 bits/sec, Record time: 2019-05-14 03:38:09
Output peak rate 150264840 bits/sec, Record time: 2019-05-14 08:22:19
Input : 12528535003628 bytes, 11657621574 packets
Output: 93571703978 bytes, 555304921 packets
Input:
Unicast: 11657620879, Multicast: 695
Broadcast: 0, Jumbo: 0
Discard: 0, Frames: 0
Pause: 0
Total Error: 0
CRC: 0, Giants: 0
Jabbers: 0, Fragments: 0
Runts: 0, DropEvents: 0
Alignments: 0, Symbols: 0
Ignoreds: 0
Output:
Unicast: 536390526, Multicast: 695
Broadcast: 0, Jumbo: 0
Discard: 0, Buffers Purged: 0
Pause: 18913700
Input bandwidth utilization threshold : 90.00%
Output bandwidth utilization threshold: 90.00%
Last 10 seconds input utility rate: 8.42%
Last 10 seconds output utility rate: 0.05%
Upstream Packet Loss
<HUAWEI> display qos buffer ingress-statistics interface 100GE1/0/1
Interface Dropped Drop Rate Drop Time
(Packets/Bytes) (pps/bps)
----------------------------------------------------------------
100GE1/0/1 0 0 -
0 0
----------------------------------------------------------------
- The headroom buffer space is small.
In a majority of cases, the automatic buffer optimization function allocates a proper headroom buffer for lossless queues in the inbound direction of an interface. The headroom buffer space ensures that no packet loss occurs in the period in which PFC takes effect. Therefore, if packet loss occurs in the inbound direction of an interface, run the display qos buffer ingress-usage command to check whether the port- and chip-level headroom buffer spaces for lossless queues are proper.
<HUAWEI> display qos buffer ingress-usage interface 100ge1/0/1 Ingress Buffer Usage (KBytes) on lossless priority: (Current/Total) *: Dynamic threshold -------------------------------------------------------------------------- Interface Priority Guaranteed PFC-Xoff Headroom -------------------------------------------------------------------------- 100GE1/0/1 3 0/2 0/4* 0/67 -------------------------------------------------------------------------- Service Pool: 12/30699 Headroom Pool: 0/4095 --------------------------------------------------------------------------
If the buffer space that is automatically allocated is improper, disable the VIQ function and manually configure the buffer space. (If VIQ is disabled, skip this step.)
<HUAWEI> system-view [~HUAWEI] low-latency fabric [~HUAWEI-low-latency-fabric] undo qos buffer auto-configuration enable //Disable automatic buffer optimization of lossless queues. [*HUAWEI-low-latency-fabric] quit
You are advised to adjust the buffer space based on the current port- and chip-level headroom buffer spaces (information in bold in the preceding display qos buffer ingress-usage command output). It is recommended that the headroom buffer of the PFC priority of the interface be adjusted in the unit of 8 Kbytes and the global headroom pool buffer space of the forwarding chip be adjusted in the unit of 128 KBytes.<HUAWEI> system-view [~HUAWEI] interface 100ge 1/0/1 [*HUAWEI-100GE1/0/1] dcb pfc buffer 3 xoff dynamic 4 hdrm 75 kbytes //Increase the headroom buffer space by 8 Kbytes. [*HUAWEI-100GE1/0/1] quit [*HUAWEI] qos buffer headroom-pool size 4223 kbytes slot 1 //Increase the headroom pool buffer space in the chip by 128 Kbytes. [*HUAWEI] commit
Recommended settings:
Table 2-11 lists the recommended headroom buffer spaces on an interface.
Table 2-11 Recommended headroom buffer spaces on an interfaceInterconnection Models
Port Type/Location
Headroom Buffer Space
CE6865-48S8CQ-EI and CE8850-64CQ-EI
25GE
250 cells
100GE
330 cells
CE6865-48S8CQ-EI/CE8850-64CQ-EI and CloudEngine 16800
Ports on CE6865-48S8CQ-EI
700 cells
Ports on CE8850-64CQ-EI
700 cells
Ports on CloudEngine 16800
2000 cells
- The PFC function is not enabled on the peer end.
Check whether PFC is enabled on the upstream device. If not, enable PFC on the upstream device.
Downstream Packet Loss
<HUAWEI> display qos queue statistics interface 100ge1/0/1
Queue CIR/PIR Passed Pass Rate Dropped Drop Rate Drop Time
(% or kbps) (Packets/Bytes) (pps/bps) (Packets/Bytes) (pps/bps)
----------------------------------------------------------------------------------------------
0 0 2 0 0 0 -
100000000 156 0 0 0
----------------------------------------------------------------------------------------------
1 0 0 0 0 0 -
100000000 0 0 0 0
----------------------------------------------------------------------------------------------
2 0 0 0 0 0 -
100000000 0 0 0 0
----------------------------------------------------------------------------------------------
3 0 11852030380 1081904 0 0 -
100000000 10549341258548 8232848944 0 0
----------------------------------------------------------------------------------------------
4 0 6 0 0 0 -
100000000 1092 0 0 0
----------------------------------------------------------------------------------------------
5 0 682363 12 0 0 -
100000000 148622146 19160 0 0
----------------------------------------------------------------------------------------------
6 0 29332473 3329 0 0 -
100000000 2405730349 2184528 0 0
----------------------------------------------------------------------------------------------
7 0 0 0 0 0 -
100000000 0 0 0 0
----------------------------------------------------------------------------------------------
Typically, no packet loss occurs in the outbound lossless queue corresponding to RoCEv2. If packet loss occurs, the concurrent or burst RoCEv2 traffic is heavy. For example, traffic of the Incast model is transmitted, and the current configuration fails to meet the requirements. To solve the problem, try the following methods:
- Decrease the uplink PFC threshold.
If the uplink PFC threshold is set to a large value, the downlink dynamic buffer threshold is triggered first when congestion occurs. In this case, run the dcb pfc buffer command to reduce the PFC XOFF threshold. For details, refer to Optimizing Lossless Service Performance in Scenarios Without Packet Loss. You can configure the dynamic or static PFC threshold as required.
- Increase the downlink dynamic buffer threshold of lossless queues.
If the downlink dynamic buffer of lossless queues is insufficient and incast traffic is transmitted, packet loss may occur in the downstream direction before PFC takes effect. To increase the downlink dynamic buffer of a lossless queue, perform the following operations:
- Run the display qos buffer egress-usage command to check the buffer usage of outbound queues on an interface.
<HUAWEI> display qos buffer egress-usage interface 100ge1/0/1 Egress Buffer Usage (KBytes) on single queue: (Current/Total) *: Dynamic threshold ------------------------------------------------------------ Interface Queue Type Guaranteed Shared ------------------------------------------------------------ 100GE1/0/1 0 Lossy 0/1 0/5* 1 Lossy 0/1 0/5* 2 Lossy 0/1 0/5* 3 Lossless 0/1 0/10156 4 Lossy 0/1 0/5* 5 Lossy 0/1 0/5* 6 Lossy 0/1 0/5* 7 Lossy 0/1 0/5* ------------------------------------------------------------ Lossless Service Pool (cells): 0/0 Lossy Service Pool (cells): 0/151136 ------------------------------------------------------------ - Run the qos buffer queue command to manually configure the queue-level service buffer and guaranteed buffer for outbound queues. (The guaranteed buffer cannot be shared. Therefore, adjust the service buffer.) Use either of the following methods to adjust the service buffer:
- Increase the queue-level service buffer in the outbound direction of lossless queues. If the VIQ function has been enabled, disable the VIQ function first. The detailed procedure is not described.
<HUAWEI> system-view [~HUAWEI] interface 100ge 1/0/1 [*HUAWEI-100GE1/0/1] qos buffer queue 3 shared-threshold dynamic 15 //Increase the queue-level service buffer in the outbound direction of lossless queues. [*HUAWEI-100GE1/0/1] quit [*HUAWEI] commit
- Decrease the queue-level service buffer in the outbound direction of lossy queues. The buffer size of lossy queues can be configured in scenarios where the VIQ function is enabled.
[~HUAWEI] interface 100ge 1/0/1 [*HUAWEI-100GE1/0/1] qos buffer queue 2 shared-threshold dynamic 3 //Decrease the queue-level service buffer in the outbound direction of lossy queues. [*HUAWEI-100GE1/0/1] quit [*HUAWEI] commit
Recommended settings:
To reduce the buffer occupied by lossy queues and ensure the available buffer for lossless queues, you are advised to set dynamic to 15 for lossless queues and to a value in the range from 1 to 3 for lossy queues.
- Increase the queue-level service buffer in the outbound direction of lossless queues. If the VIQ function has been enabled, disable the VIQ function first. The detailed procedure is not described.
- Run the display qos buffer egress-usage command to check the buffer usage of outbound queues on an interface.