How to Analyze IPSec Failures

ISAKMP Packet Format

Internet Security Association and Key Management Protocol (ISAKMP) is the basis of IKE. IKE uses ISAKMP packets for security association (SA) negotiation, key exchange, and peer identity verification.

Figure 1-1 shows the ISAKMP packet format.

Figure 1-1 ISAKMP packet format

IP Header

Source address: indicates the IP address that initiates IKE negotiation. It can be an interface IP address or an IP address configured using the tunnel local command.

Destination address: indicates the IP address of the remote device. It is configured using the remote-address command.

UDP Header

IKE uses port 500 to initiate negotiation and respond to negotiation. In the NAT traversal scenario, the gateway converts the detected port number 500 to 4500.

ISAKMP Header

Initiator's Cookie (SPI): specifies a number used by the initiator to uniquely identify an IKE SA. The SPI cannot be 0.

Responder's Cookie (SPI): specifies a number used by the responder to uniquely identify an IKE SA. The SPI in the first message is 0, and in later messages cannot be 0.

Next Payload: indicates the type of next payload in a message. If the current payload is the last one in a message, the value of this field is 0. This field provides the linking capability between payloads. To add a new payload to the end of a message, specify the type of the new payload at the end of the previous payload.

Table1 and Table2 list the next payload types in IKEv1 and IKEv2.

Maj Ver/Min Ver: identifies the ISAKMP version by numbers of the Maj Ver and Min Ver fields. In IKE, the two fields are combined.

Version: In IKEv1, the value of this field is 1.0. In IKEv2, the value of this field is 2.0.

Exchange Type: indicates the exchange type. This field constrains the payloads sent in each message and orderings of messages in an exchange.

Table3 and Table4 list exchange types in IKEv1 and IKEv2.

Flags: indicates specific options for ISAKMP exchange. For flags in this field, the meanings of each bit starting from the least significant bit are as follows:

• IKEv1: Bit 0 is the encryption bit. When set to 1, it means the payload is encrypted. Bit 1 is the commit bit and, if set, it ensures that encrypted material is not received prior to SA establishment. Bit 2 is the authentication bit and implies, if set, that the payload will be only authenticated and not encrypted. The other bits are not defined, and must be set to 0.
• IKEv2: Bit 3, if set to 1, indicates the initiator. Bit 4 indicates IKEv2 and must be set to 0. Bit 5, if set to 1, indicates the responder. The other bits are not defined, and must be set to 0.

Message ID: The value of this field is 0 in phase 1 and a random number generated by the initiator in phase 2. As the unique identifier of a message, it identifies the protocol status in phase 2 negotiation.

Message Length: indicates the length of the total ISAKM message, which is the header plus the payloads.

Negotiation Process in Main Mode

The main mode uses six ISAKMP messages to implement three bidirectional exchanges, as shown in Figure 1-2.

1. Policy exchange: messages (1) and (2)
2. Key information exchange: messages (3) and (4)
3. Identity and authentication information exchange: messages (5) and (6)

Figure 1-2 Negotiation process in main mode (pre-shared key authentication)

Policy Exchange

Message (1): The initiator sends a SA payload containing one or more IKE proposals to negotiate an IPSec proposal. The SA carries parameters, such as the encryption algorithm, authentication method, authentication algorithm, Diffie-Hellman (DH) group, and IKE SA lifetime.

Message (2): The responder finds the first matched IKE proposal and sends a SA payload to indicate that it accepts the negotiated IKE proposal.

Figure 1-3 Messages (1) and (2)

Key Information Exchange

Messages (3) and (4): The initiator and responder exchange DH public number (KE payload) and temporary random number (Ni, Nr). Ni and Nr are used to calculate the encryption key and authentication key.

Figure 1-4 Messages (3) and (4)

Identity and Authentication Information Exchange

Messages (5) and (6): The initiator and responder exchange the identity ID (ID payload) and verify hash values (AUTH payload). Information transmitted in messages (5) and (6) is encrypted using the key generated based on key information exchanged in messages (3) and (4), so identity information is protected. As the ID payload is encrypted in main mode, you cannot view its value in the obtained packet header.

Figure 1-5 Messages (5) and (6)

Negotiation Process in Aggressive Mode

In aggressive mode, only three messages are used in the exchange process, as shown in Figure 1-6.

Messages (1) and (2) are used to negotiate IKE proposal and exchange the Diffie-Hellman public number, mandatory auxiliary information, and identity information. Message (2) also contains the identity information sent by the responder to the initiator for authentication. Message (3) is used by the responder to authenticate the initiator.

Figure 1-6 Negotiation process in aggressive mode (pre-shared key authentication)

Message (1): The initiator sends a SA payload containing one IKE proposal. In aggressive mode, the SA payload contains only one proposal (encryption algorithm, authentication method, authentication algorithm, DH group, and IKE SA lifetime). The responder can accept or reject this proposal. This message also carries the DH public number (KE payload), temporary random number (Ni payload), and identity ID (ID payload). Identity information is not encrypted in aggressive mode, so you can view the information in the obtained packet header.

Figure 1-7 Message (1)

Message (2): The responder sends a SA payload containing the recommended IKE proposal. This message also carries the DH public number (KE payload), temporary random number (Nr payload), identity ID (ID payload), and hash value (AUTH payload).

Figure 1-8 Message (2)

Message (3): The initiator sends this message to verify the hash value and confirm negotiation success.

Figure 1-9 Message (3)

Summary

After IKEv1 phase 1 negotiation is complete, run the display ike sa command to view the IKE SA establishment. If Flag is displayed as RD or RD|ST, IKE SAs are established successfully. The value ST indicates that the local end initiates SA negotiation.

<sysname> display ike sa 
IKE SA information :  
      Conn-ID      Peer                 VPN    Flag(s)        Phase 
  ------------------------------------------------------------------ 
      16           2.1.1.1:500                 RD|ST          v1:2 
      14           2.1.1.1:500                 RD|ST          v1:1 
                                                
   Number of IKE SA : 2 
  ------------------------------------------------------------------- 
  RD--READY   ST--STAYALIVE   RL--REPLACED   FD--FADING   TO--TIMEOUT 
  HRT--HEARTBEAT   LKG--LAST KNOWN GOOD SEQ NO.   BCK--BACKED UP 
  M--ACTIVE   S--STANDBY   A--ALONE  NEG--NEGOTIATING  

If negotiation fails, the output of this command is empty, the Flag parameter is empty, or the Peer parameter is displayed as 0.0.0.0.

If parameters such as the IKE proposal and pre-shared key on two ends of a tunnel do not match, an IKE SA cannot be established. Common debugging information for inconsistent IKE parameters is as follows:

• The IKE proposals configured on two ends are different.

  <sysname> debugging ikev1 error
   Aug 17 2017 14:29:15.800.1 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:5773 Message from peer 2.1.1.1: Got NOTIFY of type NO_PROPOSAL_CHOS 
   EN
    
   Aug 17 2017 14:29:15.800.3 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:9946 Ikev1 error-info record(peer address: 2.1.1.1, error reason: ph 
   ase1 proposal mismatch,list number: 11). 

• The pre-shared keys configured on two ends are different.

  <sysname> debugging ikev1 error
   Aug 17 2017 15:18:09.800.1 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:5773 Message from peer 2.1.1.1: Got NOTIFY of type PAYLOAD_MALFORME
   D  

  <sysname> debugging ikev1 error
   Aug 17 2017 15:24:53.940.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:1053 Message from peer 1.1.1.1: Invalid Next Payload of Type 60 in  
   Payload Type 5
                                                                                    
   Aug 17 2017 15:24:53.940.3 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:6847 Message from peer 1.1.1.1: dropping Message due to notificatio 
   n type INVALID_PAYLOAD_TYPE
        
   Aug 17 2017 15:24:53.940.4 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:4538 Message from peer 1.1.1.1: Message Sort Error Occurred
                          
   Aug 17 2017 15:24:53.940.5 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:9946 Ikev1 error-info record(peer address: 1.1.1.1, error reason: ma 
   lformed payload,list number: 200). 

• The remote-address command is not configured on one end.

  <sysname> debugging ikev1 error
   Aug 17 2017 16:13:33.940.3 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:9946 Ikev1 error-info record(peer address: 1.1.1.1, error reason: pe 
   er address mismatch,list number: 200).                                           
                                                                                    
   Aug 17 2017 16:13:33.940.4 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:3595 Phase 1 Exchange: ike peer configuration not found for peer "1. 
   1.1.1"

• The remote-id parameter value on the local end differs from the name of the remote end.

  <sysname> debugging ikev1 error
   Aug 17 2017 16:25:01.850.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 25:6956 ERROR - Received remote-name(fw2) does not match with peer rem 
   ote-name(fw3)
                                                                                    
   Aug 17 2017 16:25:01.850.3 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:9946 Ikev1 error-info record(peer address: 2.1.1.1, error reason: co 
   nfig ID mismatch,list number: 148). 

Negotiation Process

Figure 1-10 shows the exchange process in quick mode.

Figure 1-10 Negotiation process in IKEv1 phase 2

Messages (1) and (2): The two messages are used to negotiate an IPSec proposal (SA payload) and negotiate the DH key group (KE payload) used in the perfect forward secrecy (PFS) function. The two messages also exchange the identity ID (ID payload, which is optional) and hash value for integrity verification (AUTH payload).

IDi and IDr are included in the ID payload, and are used to exchange the traffic selector. They ensure that data flows protected by both parties are the same. (This function is similar to that of the TS payload in IKEv2.)

Figure 1-11 Messages (1) and (2)

Message (3): The initiator sends this message to verify the hash value and confirm negotiation success. The packet format is similar to the format of message (1).

Summary

After IKEv1 phase 2 negotiation is complete, run the display ike sa command to view the IPSec SA establishment in phase 2. If Flag is displayed as RD or RD|ST, IPSec SAs are established successfully. The value ST indicates that the local end initiates SA negotiation.

<sysname> display ike sa 
IKE SA information :  
      Conn-ID      Peer                 VPN    Flag(s)        Phase 
  ------------------------------------------------------------------ 
      16           2.1.1.1:500                 RD|ST          v1:2 
      14           2.1.1.1:500                 RD|ST          v1:1 
                                                
   Number of IKE SA : 2 
  ------------------------------------------------------------------- 
  RD--READY   ST--STAYALIVE   RL--REPLACED   FD--FADING   TO--TIMEOUT 
  HRT--HEARTBEAT   LKG--LAST KNOWN GOOD SEQ NO.   BCK--BACKED UP 
  M--ACTIVE   S--STANDBY   A--ALONE  NEG--NEGOTIATING  

If negotiation fails, the Flag parameter in the command output is empty.

Inconsistent IPSec proposals, PFS algorithms, and ACL rules on two ends may lead to the IPSec SA establishment failure. Packets in phase 2 are encrypted, so you cannot check the IPSec proposal configuration by obtaining the packet headers.

Common debugging information for inconsistent IPSec parameters is as follows:

• ACL rules on two ends do not match.

  <sysname> debugging ikev1 error
   Aug 17 2017 17:35:52.350.1 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:5773 Message from peer 1.1.1.1: Got NOTIFY of type INVALID_ID_INFOR 
   MATION
                                                                                    
   Aug 17 2017 17:35:52.350.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:9946 Ikev1 error-info record(peer address: 1.1.1.1, error reason: fl 
   ow mismatch,list number: 200).  

  <sysname> debugging ipsec error
   Aug 17 2017 17:35:17.800.1 sysname IPSEC/7/IPSEC-DEBUG:  
   [IPSEC-Error] acl mismatch.(Ifindex=[7], SeqNum=[10],PeerAddress=[2.1.1.1], Peer 
   Port=[500])   

• The IPSec proposals or PFS algorithms on two ends are different.

  <sysname> debugging ikev1 error
   Aug 18 2017 09:28:45.350.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 0:9946 Ikev1 error-info record(peer address: 2.1.1.1, error reason: ph 
   ase2 proposal or pfs mismatch,list number: 200).                                 
                                                                                    
   Aug 18 2017 09:28:45.350.3 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 17:6847 Message from peer 2.1.1.1: dropping Message due to notificatio 
   n type NO_PROPOSAL_CHOSEN

  <sysname> debugging ipsec error
   Aug 18 2017 09:29:26.800.1 sysname IPSEC/7/IPSEC-DEBUG: 
   [IPSEC-Error] ipsec proposal or pfs mismatch.(Ifindex=[7], SeqNum=[10],PeerAddre 
   ss=[2.1.1.1], PeerPort=[500]) 

Negotiation Process

IKEv1 goes through two phases to establish a pair of IPSec SAs: "main mode + quick mode" or "aggressive mode + quick mode". When IKEv1 phase 1 uses the main mode, IKE peers exchange at least nine messages. When IKEv1 phase 1 uses the aggressive mode, IKE peers exchange at least six messages. IKEv2, without the main mode or aggressive mode, establishes an IKE SA and a pair of IPSec SAs by exchanging four messages at two times. Figure 1-12 shows the IKEv2 negotiation process.

Figure 1-12 IKEv2 negotiation process (pre-shared key authentication)

Messages (1) and (2): In the IKE_SA_INIT exchange, the initiator and responder negotiate an IKE proposal (SA payload) and exchange the temporary random number (Ni or Nr payload) and DH public number (KE payload).

Figure 1-13 Messages (1) and (2)

Messages (3) and (4): In the IKE_AUTH exchange, the two parties continue to exchange the identity ID (ID payload) and hash value (AUTH payload), negotiate the IPSec proposal (SA payload), and determine data flows to be protected (TS payload). The N payload is used for error announcement. After negotiation, an IKE SA and a pair of IPSec SAs are established.

Figure 1-14 Messages (3) and (4)

Summary

After IKEv2 negotiation is complete, run the display ike sa command to view the IKE SA establishment. If Flag is displayed as RD or RD|ST, IKE SAs are established successfully. The value ST indicates that the local end initiates SA negotiation.

<sysname> display ike sa 
IKE SA information :  
      Conn-ID      Peer                 VPN    Flag(s)        Phase 
  ------------------------------------------------------------------ 
      16           2.1.1.1:500                 RD|ST          v2:2 
      14           2.1.1.1:500                 RD|ST          v2:1 
                                                
   Number of IKE SA : 2 
  ------------------------------------------------------------------- 
  RD--READY   ST--STAYALIVE   RL--REPLACED   FD--FADING   TO--TIMEOUT 
  HRT--HEARTBEAT   LKG--LAST KNOWN GOOD SEQ NO.   BCK--BACKED UP 
  M--ACTIVE   S--STANDBY   A--ALONE  NEG--NEGOTIATING  

If negotiation fails, the following may occur when you run the display ike sa command:

• If phase 1 negotiation fails, the command output is empty, the Flag parameter is empty, or the Peer parameter is displayed as 0.0.0.0.
• If phase 2 negotiation fails, information about phase 1 is normally displayed; information about phase 2 is empty or the Flag parameter is empty.

Inconsistent IKE proposals, pre-shared keys, IPSec proposals, PFS algorithms, and ACL rules on two ends may lead to the IKE SA or IPSec SA establishment failure.

Common debugging information for inconsistent IPSec parameters is as follows:

• The IKE proposals configured on two ends are different.

  <sysname> debugging ikev2 error
   Aug 18 2017 14:51:00.410.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 56:8841 Ikev2 error Info record(peer address: 2.1.1.1, error reason: p 
   hase1 proposal mismatch,list number: 200).   

• The pre-shared keys configured on two ends are different.

  <sysname> debugging ikev2 error
   Aug 18 2017 15:02:25.540.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 63:4087 Authentication failed for the peer 2.1.1.1
    
   Aug 18 2017 15:02:25.540.4 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 56:8841 Ikev2 error Info record(peer address: 2.1.1.1, error reason: a 
   uthentication fail,list number: 4). 

• The remote-id parameter value on the local end differs from the name of the remote end.

  <sysname> debugging ikev2 error
   Aug 18 2017 15:39:32.50.4 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 27:27568 ERROR - Peer remote-name(fw3) does not match with
                                                                                    
   Aug 18 2017 15:39:32.50.5 FW1 IKE/7/IKE_Debug: 
   IKE_ERROR 56:8841 Ikev2 error Info record(peer address: 2.1.1.1, error reason: c 
   onfig ID mismatch,list number: 151). 

• The remote-address command is not configured on one end.

  <sysname> debugging ikev2 error
   Aug 18 2017 15:23:04.400.1 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 56:8841 Ikev2 error Info record(peer address: 2.1.1.1, error reason: p 
   eer address mismatch,list number: 107).  

• ACL rules on two ends do not match.

  <sysname> debugging ikev2 error
   Aug 18 2017 15:27:40.750.1 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 56:8841 Ikev2 error Info record(peer address: 1.1.1.1, error reason: f 
   low mismatch,list number: 200).  

  <sysname> debugging ipsec error
   Aug 18 2017 15:29:46.650.1 sysname IPSEC/7/IPSEC-DEBUG: 
   [IPSEC-Error] acl mismatch.(Ifindex=[7], SeqNum=[0],PeerAddress=[1.1.1.1], PeerP 
   ort=[500]) 

• The IPSec proposals or PFS algorithms on two ends are different.

  <sysname> debugging ikev2 error
   Aug 19 2017 09:41:40.800.2 sysname IKE/7/IKE_Debug: 
   IKE_ERROR 56:8841 Ikev2 error Info record(peer address: 2.1.1.1, error reason: p 
   hase2 proposal or pfs mismatch,list number: 200).    

  <sysname> debugging ipsec error
   Aug 19 2017 09:44:25.760.2 sysname IPSEC/7/IPSEC-DEBUG:                              
   [IPSEC-Error] ipsec proposal or pfs mismatch.(Ifindex=[7], SeqNum=[10],PeerAddre 
   ss=[2.1.1.1], PeerPort=[500])  

IKEv1 Negotiation with NAT Traversal

In the NAT traversal scenario using IKEv1, the following must be detected in phase 1 negotiation:

• Support for NAT traversal (NAT-T capability)

  IKEv1 uses the VID payload to determine whether NAT traversal is supported.

• Presence of a NAT gateway

  IKEv1 uses the NAT-D payload to detect whether a NAT gateway exists between two IKE peers and determine its location (finding the device connecting to the NAT gateway) if available.

As the NAT gateway changes the source port number in IKE UDP packets, the responder must be able to process IKE packets whose source port number is not 500.

Figure 1-15 shows the negotiation process of IKEv1 phase 1 in aggressive mode with NAT traversal.

The negotiation process in main mode with NAT traversal is omitted here. The process is similar to Negotiation Process in Main Mode, with only the following differences: The VID payload is added to messages (1) and (2), and the NAT-D payload is added to messages (3) and (4).

Figure 1-15 Negotiation process of IKEv1 phase 1 in aggressive mode with NAT traversal (pre-shared key authentication)

Message (1): The initiator inserts the VID payload to an IKE message to inform the receiver that it supports NAT traversal. If IKEv1 messages sent by both parties contain the VID payload, both parties support NAT-T. In this case, negotiation continues. Otherwise, the process comes to an end.

Figure 1-16 Message (1)

Message (2): The responder inserts the VID payload to an IKE message to inform the receiver that it supports NAT traversal. The responder also adds two NAT-D payloads to the message. The first NAT-D payload contains the IP address of the IKE peer and the port's hash value; the second NAT-D payload contains the IP address of the local end and the port's hash value. Both the initiator and responder calculate the hash values and compare them with local ones. The device with inconsistent hash values is connected to the NAT gateway.

Figure 1-17 Message (2)

Message (3): After confirming NAT-T support and detecting NAT gateway, the port number in subsequent UDP packets is modified to 4500 if a NAT gateway is detected.

Figure 1-18 Message (3)

The non-ESP marker field (four all-0 bytes) is added under UDP encapsulation of IPSec packets to distinguish from encapsulation of ESP packets.

In IKEv1 phase 2 SA negotiation, whether NAT traversal is used and the encapsulation mode (UDP-Encapsulated-tunnel or UDP-Encapsulated-transport) for NAT traversal must be confirmed. Then, subsequently transmitted ESP packets are encapsulated using UDP (as the AH protocol does not support NAT traversal). After an ESP packet is encapsulated using UDP, the packet does not contain the non-ESP marker field. Instead, this field is replaced by the SPI field, which consists non-0 bytes.

IKEv2 Negotiation with NAT Traversal

In the NAT traversal scenario using IKEv2, the initiator and responder add two N payloads (next to the Ni or Nr payload) in IKE_SA_INIT exchange. One message type is NAT_DETECTION_SOURCE_IP, which identifies the initiator's IP address. The other message type is NAT_DETECTION_DESTINATION_IP, which identifies the responder's IP address.

Figure 1-19 shows the IKEv2 negotiation process with NAT traversal.

Figure 1-19 IKEv2 negotiation process with NAT traversal (pre-shared key authentication)

Messages (1) and (2): Two N payloads are inserted to an IKE message. The first N payload contains the IP address of the local end and the port's hash value; the second N payload contains the IP address of the IKE peer and the port's hash value. Both the initiator and responder calculate the hash values and compare them with local ones. The device with inconsistent hash values is connected to the NAT gateway.

Figure 1-20 Messages (1) and (2)

Messages (3) and (4): After the IKE_SA_INIT exchange, the port number of UDP packets is modified to 4500 if a NAT device is detected.

Figure 1-21 Messages (3) and (4)

Summary

The NAT traversal function must be enabled on both ends to ensure IKE negotiation success.

After IKE negotiation is complete, run the display ike sa command to view the SA establishment. If Flag is displayed as RD or RD|ST, IKE SAs are established successfully. The value ST indicates that the local end initiates SA negotiation.

<sysname> display ike sa 
IKE SA information :  
      Conn-ID      Peer                 VPN    Flag(s)        Phase 
  ------------------------------------------------------------------ 
      16           2.1.1.1:4500                RD|ST          v2:2 
      14           2.1.1.1:4500                RD|ST          v2:1 
                                                
   Number of IKE SA : 2 
  ------------------------------------------------------------------- 
  RD--READY   ST--STAYALIVE   RL--REPLACED   FD--FADING   TO--TIMEOUT 
  HRT--HEARTBEAT   LKG--LAST KNOWN GOOD SEQ NO.   BCK--BACKED UP 
  M--ACTIVE   S--STANDBY   A--ALONE  NEG--NEGOTIATING  

Impact by CPU Performance

• IPSec service quality deterioration caused by CPU resource consumption during packet fragmentation
  • By maximum transmission unit (MTU) value

    The MTU value supported by a link is determined by the smallest MTU value of an interface on this link, while the interface MTU value varies with the interface type. (For example, the default MTU value of an Ethernet interface is 1500 bytes.) If the size of a packet exceeds the interface MTU, the device fragments the encrypted packet before sending it. The receiver needs to reassemble all the fragments of an IP packet before decrypting it. Fragmentation and reassembly consume CPU resources.

    In the IPSec packet encapsulation process, the cost increases each time the IPSec module encapsulates the received original IP packets. The increased cost varies with the encapsulation protocol. For details, see Table1. Assume that the cost increases by 80 bytes after IPSec processing. Packets containing more than 1420 bytes will carry more than 1500 bytes after IPSec encapsulation; therefore, they need to be fragmented. When most packets in data flows are large packets exceeding 1420 bytes, a large number of CPU resources are consumed. This will lead to decrease of the IPSec service access speed and service quality deterioration.

    Table 1-5 Cost of bytes by protocol

    Protocol	Increased Bytes
    ESP	Default value: 56 The increased cost of ESP depends on the encryption algorithm and whether an authentication algorithm is used.
    AH	24
    GRE	24
    NAT-T	8
    L2TP	12
    PPPoE	8
    IPSec tunnel mode	20
    TCP	8

• By Transmission Control Protocol maximum segment size (TCP MSS)

  If the total packet length (MSS plus various types of costs, including the TCP packet header, IP packet header, and IPSec packet header) exceeds the link MTU, data packets will be fragmented. Fragmentation will lead to consumption of more CPU resources. In addition, encryption and decryption of packet fragments also consume CPU resources of devices on the transmission link. Too much CPU resource consumption will lead to data packet loss.

  Some upper layer applications (such as the application layer protocol HTTP) will set the Don't Fragment (DF) flag of IP packets to valid to prevent TCP packet fragmentation. If the DF flag is set to valid but the interface MTU is smaller than the MSS, the device will discard packets because it cannot forcibly fragment the TCP packets.

  • IPSec service quality deterioration caused by CPU resource consumption by other features

    Features such as IPSec, content security, and attack defense consume large CPU resources. Therefore, if IPSec, content security, and other features are enabled simultaneously, the IPSec service access speed will become slow. The larger the data volume, the more obvious the problem.

Impact by Internet Quality

IPSec is used to secure communication between virtual networks over the Internet. The transmission quality of the Internet has a direct import on the IPSec service quality. You can deploy the VPN in bypass mode to determine whether the poor IPSec quality is caused by the poor transmission quality on the Internet. If the transmission quality on the Internet is poor, contact the carrier. The Internet may encounter the following problems:

• The Internet encounters network congestion, leading to packet loss or network interruption. Packet loss has a huge impact on the quality of voice and video services, and obviously slows down web browsing and file transfer. Network interruption will directly lead to service interruption.
• The carrier denies packets of specific types, for example, UDP packets. If so, IKE negotiation will fail in the NAT traversal scenario.
• The carrier disables ports such as ports 500 and 4500 used by the IPSec service. If port 500 is disabled, IKE negotiation will fail. If port 4500 is disabled, IKE negotiation will fail in the NAT traversal scenario.