IPSec Framework
Security Association
A Security Association (SA) is an agreement between two peers on certain elements. It describes how to use security services (such as encryption) between peers for secure communication. The elements include the security protocol, characteristics of data flows to be protected, encapsulation mode of data transmitted between peers, encryption and authentication algorithms, and lifetimes of keys (used for secure data conversion and transmission) and SAs.
IPSec SAs need to be established for secure data transmission between IPSec peers. An IPSec SA is identified by three parameters: security parameter index (SPI), destination IP address, and security protocol number (AH or ESP). The SPI is a 32-bit value generated to uniquely identify an SA, and is encapsulated in an AH or ESP header.
IPSec SAs are unidirectional and are generally established in pairs (inbound and outbound). Therefore, at least one pair of IPSec SAs are established between two IPSec peers to form a secure IPSec tunnel that protects data flows in both directions, as shown in Figure 5-1.
In addition, the number of IPSec SAs required also depends on the security protocols used to protect traffic between two peers. If either AH or ESP is used, two SAs are required to protect incoming and outgoing traffic. If both AH and ESP are used, four SAs are required, two for each protocol.
IPSec SAs can be established manually or through the Internet Key Exchange (IKE) protocol. Table 5-1 compares the two IPSec SAs establishment modes.
| Item | Manually Establishing IPSec SAs | Automatically Establishing IPSec SAs Through IKE |
|---|---|---|
| Configuration and update of keys used for encryption and authentication | Manually configured and updated; prone to errors High key management cost |
Generated by the Diffie-Hellman (DH) algorithm and dynamically updated Low key management cost |
| SPI value | Manually configured | Randomly generated |
| Lifetime of an IPSec SA | Valid permanently | Negotiated by two peers (SAs are dynamically updated.) |
| Security | Low | High |
| Applicable scenario | Small-sized networks | Large-, medium-, and small-sized networks |
Security Protocols
AH and ESP are the two IP-based transport-layer protocols used by IPSec to provide security services such as authentication and encryption.
AH
AH is used to authenticate, but not encrypt, IP traffic. An AH header is appended to the standard IP header in each data packet, as described in Encapsulation Modes. AH provides data origin authentication and data integrity check that is performed on an entire IP packet. The sender performs hash calculation on data packets and the authentication key. Upon receipt of data packets carrying the calculation result, the receiver also performs hash calculation and compares the calculation result with that received from the sender. Any changes to the data during transmission will invalidate the calculation result.
ESP
ESP provides encryption and optional authentication. An ESP header is appended to the standard IP header in each data packet, and an ESP trailer and ESP Authentication Data are appended to each data packet, as described in Encapsulation Modes. ESP encrypts the IP payload and then encapsulates it into a data packet to ensure data confidentiality. ESP protects the IP header only in tunnel mode.
Table 5-2 compares AH and ESP.
| Security Feature | AH | ESP |
|---|---|---|
| Protocol number | 51 | 50 |
| Data integrity check | Supported (checking the entire IP packet) | Supported (not checking the IP header in transport mode; checking the entire IP packet in tunnel mode) |
| Data origin authentication | Supported | Supported |
| Data encryption | Not supported | Supported |
| Anti-replay | Supported | Supported |
| IPSec NAT traversal (NAT-T) | Not supported | Supported |
AH and ESP can be used together when high security is required.
AH Header and ESP Header
AH header
Figure 5-2 shows the format of an AH header, and Table 5-3 describes fields in the AH header.
Field |
Length |
Description |
|---|---|---|
Next Header |
8 bits |
This field identifies the type of the payload following the AH header. In transport mode, the Next Header field is the number of the protected upper-layer protocol (TCP or UDP) or ESP. In tunnel mode, the Next Header field is the number of the IP or ESP protocol. NOTE:
When AH and ESP are used together, the Next Header following an AH header is an ESP header. |
Payload Len |
8 bits |
This field specifies the length of the AH header in 32-bit words (4-byte units) minus 2. The default value is 4. |
Reserved |
16 bits |
This field is reserved for future use and defaults to 0. |
SPI |
32 bits |
This field uniquely identifies an IPSec SA. |
Sequence Number |
32 bits |
This field is a counter that monotonically increments from 1. It uniquely identifies a packet to prevent replay attacks. |
Authentication Data |
Integral multiple of 32 bits. It is 96 bits in common cases. |
This field contains the result of Integrity Check Value (ICV) calculation, which is used by the receiver for data integrity check. Authentication algorithms include MD5, SHA1, and SHA2. NOTE:
The MD5 and SHA1 authentication algorithms have security risks. The SHA2 algorithm is recommended. |
ESP header
Figure 5-3 shows the format of an ESP header, and Table 5-4 describes fields in the ESP header.
Field |
Length |
Description |
|---|---|---|
SPI |
32 bits |
This field uniquely identifies an IPSec SA. |
Sequence Number |
32 bits |
This field is a counter that monotonically increments from 1. It uniquely identifies a packet to prevent replay attacks. |
Payload data |
- |
This field contains the protected variable-length data content in the original IP packet. The type of content protected by ESP is identified by the Next Header field. |
Padding |
- |
This field extends the payload data to a size that fits the encryption's cipher block size. The Padding field length depends on the payload data length and algorithm. |
Pad Length |
8 bits |
This field specifies the length of the Padding field. The value 0 indicates no padding. |
Next Header |
8 bits |
This field identifies the type of the payload following the ESP header. In transport mode, the Next Header field is the number of the protected upper-layer protocol (TCP or UDP). In tunnel mode, the Next Header field is the number of the IP protocol. |
Authentication Data |
Integral multiple of 32 bits. It is 96 bits in common cases. |
This field contains the result of ICV calculation, which is used by the receiver for data integrity check. Authentication algorithms are the same as those of AH. The authentication function of ESP is optional. If data check is enabled, an ICV value is appended to encrypted data. |
Encapsulation Modes
Encapsulation is a process of adding AH or ESP fields to original IP packets for packet authentication and encryption. This process is implemented in either transport or tunnel mode.
Transport Mode
IPSec transport mode works by inserting an AH or ESP header between an IP header and a transport-layer protocol header to protect the TCP, UDP, or ICMP payload. Because no additional IP header is added, IP addresses in the original packets are visible in the IP header of the post-encrypted packet. Figure 5-4 shows an example of TCP packet encapsulation in transport mode.
In transport mode, AH protects the IP header, but ESP does not.
Tunnel Mode
IPSec tunnel mode works by encrypting and authenticating an entire IP packet, including the IP header and payload. In this mode, an AH or ESP header is added before the raw IP header, and a new IP header is added before the AH or ESP header. Figure 5-5 shows an example of TCP packet encapsulation in tunnel mode.
In tunnel mode, AH protects the new IP header, but ESP does not.
Comparisons Between the Transport Mode and Tunnel Mode
The main differences between the transport mode and tunnel mode are as follows:
The tunnel mode is more secure because original IP packets are completely authenticated and encrypted. This mode hides the IP address, protocol type, and port number in an original IP packet.
The tunnel mode generates an additional IP header, occupying more bandwidth than the transport mode.
The transport mode is mainly used for communication between two hosts or between a host and a VPN gateway. The tunnel mode is mainly used for communication between two VPN gateways or between a host and a VPN gateway.
When both AH and ESP are used to protect traffic, they must use the same encapsulation mode.
Encryption and Authentication
IPSec provides two security mechanisms: encryption and authentication. The encryption mechanism ensures data confidentiality and prevents data from being intercepted during transmission. The authentication mechanism ensures data integrity and reliability and prevents data from being forged or tampered with during transmission.
Encryption
IPSec uses symmetric encryption algorithms to encrypt and decrypt data. Figure 5-6 shows the process of encrypting and decrypting data through the same key (that is, a symmetric key) between two IPSec peers.
A symmetric key used for encryption and decryption can be configured manually or generated automatically through IKE negotiation.
Common symmetric encryption algorithms include Data Encryption Standard (DES), Triple Data Encryption Standard (3DES), and Advanced Encryption Standard (AES). DES and 3DES are not recommended because they are insecure and pose security risks.
Authentication
IPSec uses the Keyed-Hash Message Authentication Code (HMAC) function to compare Integrity Check Value (ICV) to check integrity and authenticity of data packets.
Encryption and authentication are often used together. As shown in Figure 5-7, the IPSec sender encrypts an IP packet, generates an Integrity Check Value (ICV) through an authentication algorithm and a symmetric key, and then sends both the encrypted IP packet and Integrity Check Value (ICV) to the IPSec receiver. The IPSec receiver processes the encrypted packet using the same authentication algorithm and symmetric key, and compares the received Integrity Check Value (ICV) with the locally generated one to check integrity and authenticity of data in the received IP packet. The IPSec receiver discards any packets that fail the authentication and decrypts only those that pass the authentication.
Similar to a symmetric key used for encryption, a symmetric key used for authentication can be configured manually or generated automatically through IKE Protocol negotiation.
Common authentication algorithms include Message Digest 5 (MD5), Secure Hash Algorithm 1 (SHA1), and SHA2. MD5 and SHA1 are not recommended because they are insecure and pose security risks.
Key Exchange
How to securely share a symmetric key is an important issue during encryption and authentication using the key. Two methods are available to address this issue:
Out-of-band key sharing
An encryption key and an authentication key are manually configured on the IPSec sender and receiver. The two parties ensure key consistency in out-of-band mode, for example, by exchanging phone calls or emails. This mode not only has poor security and scalability, but also multiplies the workload in configuring keys in point-to-multipoint networking. In addition, this mode is difficult to implement because the keys need to be modified periodically for security purposes.
Using a secure key distribution protocol
Keys are generated through IKE negotiation. The IKE protocol uses the DH algorithm to implement secure key distribution over an insecure network. This mode is easy to configure and provides high scalability, especially on a large dynamic network. Two communicating parties exchange keying materials to calculate shared keys. Even if a third party obtains all the exchanged data used to calculate the shared keys, it cannot calculate the shared keys. Therefore, this mode greatly improves security.
IKE Protocol
IKE is a UDP-based application-layer protocol built on the Internet Security Association and Key Management Protocol (ISAKMP) framework. IPsec uses the IKE protocol for key auto-negotiation and IPSec SA establishment, simplifying IPSec configuration and maintenance.
Figure 5-8 shows the relationship between IKE and IPSec. Two peers establish an IKE SA for identity authentication and key exchange. Protected by the IKE SA, the peers negotiate a pair of IPSec SAs using the AH or ESP protocol and other configured parameters. Subsequently, data is encrypted and transmitted between the peers in an IPSec tunnel.
An IKE SA is bidirectional, so only one IKE SA needs to be established between two peers.
IKE security mechanisms
Identity authentication
Two peers authenticate the identity (IP address or name) of each other, using pre-shared key (PSK) authentication or Rivest-Shamir-Adleman (RSA) signature authentication.
- PSK authentication: Two peers compute the hash value of packets using the PSK and check whether they obtain the same hash value. If the peers obtain the same hash value, the authentication succeeds. Otherwise, authentication fails. For a peer that has multiple remote IPSec peers, PSK authentication requires each pair of peers to have the same PSK. This authentication method can be easily implemented on small-scale networks but has low security.
- RSA signature authentication: Two peers use a certificate issued by a certificate authority (CA) to verify the validity of a digital certificate. Each peer has a public key (transmitted over the network) and a private key (possessed by itself). The sender computes a hash value for original packets, and then encrypts the hash value using its private key to generate a digital signature. The receiver decrypts the digital signature using the public key received from the sender, and then computes a hash value for packets. If the computed hash value is the same as that decrypted from the digital signature, the authentication succeeds. Otherwise, authentication fails. RSA signature authentication provides high security but requires digital certificates issued by a CA. This authentication method is applicable to large-scale networks.
IKE supports the following authentication algorithms: MD5, SHA1, SHA2-256, SHA2-384, and SHA2-512.
Identity protection
After a key is generated, identity data is encrypted using the key to ensure secure transmission.
IKE supports the following encryption algorithms: DES, 3DES, AES-128, AES-192, and AES-256.
DH
DH is a public key exchange method that generates keying materials and uses ISAKMP messages to exchange keying materials between the sender and receiver. The two devices compute the same symmetric key and generate an encryption key and an authentication key based on this symmetric key. The encryption key and authentication key are never exchanged between the two devices. DH key exchange is central to IKE.
Perfect Forward Secrecy (PFS)
If PFS is enabled, an additional DH key exchange is performed when keys used in IPSec SAs are generated based on the keys used in the IKE SA. This ensures that the keys used in IPSec SAs remain indecipherable even if a key used in the IKE SA is deciphered.
- The MD5 and SHA1 authentication algorithms are insecure. The more secure SHA2-256, SHA2-384, or SHA2-512 algorithm is recommended.
The DES and 3DES encryption algorithms are insecure. The more secure AES algorithm is recommended.
IKE versions
IKE has two versions: IKEv1 and IKEv2. IKEv2 has the following advantages over IKEv1:
Simplifies SA negotiation and improves negotiation efficiency.
IKEv1 goes through two phases to negotiate the key and establish IPSec SAs. In phase 1, two peers negotiate and establish a secure tunnel, which is an IKE SA. In phase 2, the peers establish a pair of IPSec SAs through the secure channel established in phase 1. IKEv2 generates the key and establishes IPSec SAs in just one negotiation, simplifying the negotiation process. For details, see Establishing SAs Through IKEv1 Negotiation and Establishing SAs Through IKEv2 Negotiation.
Fixes many cryptographic vulnerabilities, enhancing security.