Introduction

Ethernet virtual private network (EVPN) is a VPN technology used for Layer 2 internetworking. EVPN is similar to BGP/MPLS IP VPN. EVPN defines a new type of BGP network layer reachability information (NLRI), called the EVPN NLRI. The EVPN NLRI defines new BGP EVPN routes to implement MAC address learning and advertisement between Layer 2 networks at different sites.

VXLAN does not provide a control plane, and VTEP discovery and host information (IP and MAC addresses, VNIs, and gateway VTEP IP address) learning are implemented by traffic flooding on the data plane, resulting in high traffic volumes on DC networks. To address this problem, VXLAN uses EVPN as the control plane. EVPN allows VTEPs to exchange BGP EVPN routes to implement automatic VTEP discovery and host information advertisement, preventing unnecessary traffic flooding.

In summary, EVPN introduces several new types of BGP EVPN routes through BGP extension to advertise VTEP addresses and host information. In this way, EVPN applied to VXLAN networks enables VTEP discovery and host information learning on the control plane instead of on the data plane.

BGP EVPN Routes

EVPN NLRI defines the following BGP EVPN route types applicable to the VXLAN control plane:

Type 2 Route: MAC/IP Route

Figure 1-1026 shows the format of a MAC/IP route.

Figure 1-1026 Format of a MAC/IP route

Table 1-467 describes the meaning of each field.

MAC/IP routes function as follows on the VXLAN control plane:

• MAC address advertisement

  To implement Layer 2 communication between intra-subnet hosts, the source and remote VTEPs must learn the MAC addresses of the hosts. The VTEPs function as BGP EVPN peers to exchange MAC/IP routes so that they can obtain the host MAC addresses. The MAC Address field identifies the MAC address of a host.

• ARP advertisement

  A MAC/IP route can carry both the MAC and IP addresses of a host, and therefore can be used to advertise ARP entries between VTEPs. The MAC Address field identifies the MAC address of the host, whereas the IP Address field identifies the IP address of the host. This type of MAC/IP route is called the ARP route.

• IP route advertisement

  In distributed VXLAN gateway scenarios, to implement Layer 3 communication between inter-subnet hosts, the source and remote VTEPs that function as Layer 3 gateways must learn the host IP routes. The VTEPs function as BGP EVPN peers to exchange MAC/IP routes so that they can obtain the host IP routes. The IP Address field identifies the destination address of the IP route. In addition, the MPLS Label2 field must carry the L3VNI. This type of MAC/IP route is called the integrated routing and bridging (IRB) route.

  An ARP route carries host MAC and IP addresses and an L2VNI. An IRB route carries host MAC and IP addresses, an L2VNI, and an L3VNI. Therefore, IRB routes carry ARP routes and can be used to advertise IP routes as well as ARP entries.

• Host IPv6 route advertisement

  In a distributed gateway scenario, to implement Layer 3 communication between hosts on different subnets, the VTEPs (functioning as Layer 3 gateways) must learn host IPv6 routes from each other. To achieve this, VTEPs functioning as BGP EVPN peers exchange MAC/IP routes to advertise host IPv6 routes to each other. The IP Address field carried in the MAC/IP routes indicates the destination addresses of host IPv6 routes, and the MPLS Label2 field must carry an L3VNI. MAC/IP routes in this case are also called IRBv6 routes.

  An ND route carries host MAC and IPv6 addresses and an L2VNI. An IRBv6 route carries host MAC and IPv6 addresses, an L2VNI, and an L3VNI. Therefore, IRBv6 routes carry ND routes and can be used to advertise both host IPv6 routes and ND entries.

Type 3 Route: Inclusive Multicast Route

An inclusive multicast route comprises a prefix and a PMSI attribute. Figure 1-1027 shows the format of an inclusive multicast route.

Figure 1-1027 Format of an inclusive multicast route

Table 1-468 describes the meaning of each field.

Inclusive multicast routes are used on the VXLAN control plane for automatic VTEP discovery and dynamic VXLAN tunnel establishment. VTEPs that function as BGP EVPN peers transmit L2VNIs and VTEPs' IP addresses through inclusive multicast routes. The originating router's IP Address field identifies the local VTEP's IP address; the MPLS Label field identifies an L2VNI. If the remote VTEP's IP address is reachable at Layer 3, a VXLAN tunnel to the remote VTEP is established. In addition, the local end creates a VNI-based ingress replication list and adds the peer VTEP IP address to the list for subsequent BUM packet forwarding.

Type 5 Route: IP Prefix Route

Figure 1-1028 shows the format of an IP prefix route.

Figure 1-1028 Format of an IP prefix route

Table 1-469 describes the meaning of each field.

An IP prefix route can carry either a host IP address or a network segment address.

• When carrying a host IP address, the route is used for IP route advertisement in distributed VXLAN gateway scenarios, which functions the same as an IRB route on the VXLAN control plane.

• When carrying a network segment address, the route can be advertised to allow hosts on a VXLAN network to access the specified network segment or external network.

Centralized VXLAN Gateway Mode

In this mode, Layer 3 gateways are configured on one device. On the network shown in Figure 1-1029, traffic across network segments is forwarded through Layer 3 gateways to implement centralized traffic management.

Figure 1-1029 Centralized VXLAN gateway networking

Distributed VXLAN Gateway Mode

Deploying distributed VXLAN gateways addresses problems that occur in centralized VXLAN gateway networking. Distributed VXLAN gateways use the spine-leaf network. In this networking, leaf nodes, which can function as Layer 3 VXLAN gateways, are used as VTEPs to establish VXLAN tunnels. Spine nodes are unaware of the VXLAN tunnels and only forward VXLAN packets between different leaf nodes. On the network shown in Figure 1-1030, Server 1 and Server 2 on different network segments both connect to Leaf 1. When Server 1 and Server 2 communicate, traffic is forwarded only through Leaf 1, not through any spine node.

Figure 1-1030 Distributed VXLAN gateway networking

A spine node supports high-speed IP forwarding capabilities.

VXLAN Tunnel Establishment

A VXLAN tunnel is identified by a pair of VTEP IP addresses. A VXLAN tunnel can be statically created after you configure local and remote VNIs, VTEP IP addresses, and an ingress replication list, and the tunnel goes Up when the pair of VTEPs is reachable at Layer 3.

On the network shown in Figure 1-1031, Leaf 1 connects to Host 1 and Host 3; Leaf 2 connects to Host 2; Spine functions as a Layer 3 gateway.

• To allow Host 3 and Host 2 to communicate, Layer 2 VNIs and an ingress replication list must be configured on Leaf 1 and Leaf 2. The peer VTEPs' IP addresses must be specified in the ingress replication list. A VXLAN tunnel can be established between Leaf 1 and Leaf 2 if their VTEPs have Layer 3 routes to each other.

• To allow Host 1 and Host 2 to communicate, Layer 2 VNIs and an ingress replication list must be configured on Leaf 1, Leaf 2, and also Spine. The peer VTEPs' IP addresses must be specified in the ingress replication list. A VXLAN tunnel can be established between Leaf 1 and Spine and between Leaf 2 and Spine if they have Layer 3 routes to the IP addresses of the VTEPs of each other.

  Although Host 1 and Host 3 both connect to Leaf 1, they belong to different subnets and must communicate through the Layer 3 gateway (Spine). Therefore, a VXLAN tunnel is also required between Leaf 1 and Spine.

Figure 1-1031 VXLAN tunnel networking

Dynamic MAC Address Learning

VXLAN supports dynamic MAC address learning to allow communication between tenants. MAC address entries are dynamically created and do not need to be manually maintained, greatly reducing maintenance workload. The following example illustrates dynamic MAC address learning for intra-subnet communication on the network shown in Figure 1-1032.

Figure 1-1032 Dynamic MAC Address Learning

1. Host 3 sends an ARP request for Host 2's MAC address. The ARP request carries the source MAC address being MAC3, destination MAC address being all Fs, source IP address being IP3, and destination IP address being IP2.

2. Upon receipt of the ARP request, Leaf 1 determines that the Layer 2 sub-interface receiving the ARP request belongs to a BD that has been bound to a VNI (20), meaning that the ARP request packet must be transmitted over the VXLAN tunnel identified by VNI 20. Leaf 1 then learns the mapping between Host 3's MAC address, BDID (Layer 2 broadcast domain ID), and inbound interface (Port1 for the Layer 2 sub-interface) that has received the ARP request and generates a MAC address entry for Host 3. The MAC address entry's outbound interface is Port1.

3. Leaf 1 then performs VXLAN encapsulation on the ARP request, with the VNI being the one bound to the BD, source IP address in the outer IP header being the VTEP's IP address of Leaf 1, destination IP address in the outer IP header being the VTEP's IP address of Leaf 2, source MAC address in the outer Ethernet header being NVE1's MAC address of Leaf 1, and destination MAC address in the outer Ethernet header being the MAC address of the next hop pointing to the destination IP address. Figure 1-1033 shows the VXLAN packet format. The VXLAN packet is then transmitted over the IP network based on the IP and MAC addresses in the outer headers and finally reaches Leaf 2.

   Figure 1-1033 VXLAN packet format
   

4. After Leaf 2 receives the VXLAN packet, it decapsulates the packet and obtains the ARP request originated from Host 3. Leaf 2 then learns the mapping between Host 3's MAC address, BDID, and VTEP's IP address of Leaf 1 and generates a MAC address entry for Host 3. Based on the next hop (VTEP's IP address of Leaf 1), the MAC address entry's outbound interface recurses to the VXLAN tunnel destined for Leaf1.

5. Leaf 2 broadcasts the ARP request in the Layer 2 domain. Upon receipt of the ARP request, Host 2 finds that the destination IP address is its own IP address and saves Host 3's MAC address to the local MAC address table. Host 2 then responds with an ARP reply.

So far, Host 2 has learned Host 3's MAC address. Therefore, Host 2 responds with a unicast ARP reply. The ARP reply is transmitted to Host 3 in the same manner. After Host 2 and Host 3 learn the MAC address of each other, they will subsequently communicate with each other in unicast mode.

Dynamic MAC address learning is required only between hosts and Layer 3 gateways in inter-subnet communication scenarios. The process is the same as that for intra-subnet communication.

Intra-Subnet Known Unicast Packet Forwarding

Intra-subnet known unicast packets are forwarded only through Layer 2 VXLAN gateways and are unknown to Layer 3 VXLAN gateways. Figure 1-1034 shows the intra-subnet known unicast packet forwarding process.

Figure 1-1034 Intra-subnet known unicast packet forwarding

1. After Leaf 1 receives Host 3's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information and searches for the outbound interface and encapsulation information in the BD.
2. Leaf 1's VTEP performs VXLAN encapsulation based on the encapsulation information obtained and forwards the packets through the outbound interface obtained.
3. Upon receipt of the VXLAN packet, Leaf 2's VTEP verifies the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. Leaf 2 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
4. Leaf 2 obtains the destination MAC address of the inner Layer 2 packet, adds VLAN tags to the packets based on the outbound interface and encapsulation information in the local MAC address table, and forwards the packets to Host 2.

Host 2 sends packets to Host 3 in the same manner.

Intra-Subnet BUM Packet Forwarding

Intra-subnet BUM packet forwarding is completed between Layer 2 VXLAN gateways in ingress replication mode. Layer 3 VXLAN gateways do not need to be aware of the process. In ingress replication mode, when a BUM packet enters a VXLAN tunnel, the ingress VTEP uses ingress replication to perform VXLAN encapsulation and send a copy of the BUM packet to every egress VTEP in the list. When the BUM packet leaves the VXLAN tunnel, the egress VTEP decapsulates the BUM packet. Figure 1-1035 shows the BUM packet forwarding process.

Figure 1-1035 Ingress replication for forwarding BUM packets

1. After Leaf 1 receives Terminal A's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information.
2. Leaf 1's VTEP obtains the ingress replication list for the VNI, replicates packets based on the list, and performs VXLAN encapsulation by adding outer headers. Leaf 1 then forwards the VXLAN packet through the outbound interface.
3. Upon receipt of the VXLAN packet, Leaf 2's VTEP and Leaf 3's VTEP verify the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. Leaf 2/Leaf 3 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
4. Leaf 2/Leaf 3 checks the destination MAC address of the inner Layer 2 packet and finds it a BUM MAC address. Therefore, Leaf 2/Leaf 3 broadcasts the packet onto the network connected to the terminals (not the VXLAN tunnel side) in the Layer 2 broadcast domain. Specifically, Leaf 2/Leaf 3 finds the outbound interfaces and encapsulation information not related to the VXLAN tunnel, adds VLAN tags to the packet, and forwards the packet to Terminal B/Terminal C.

Terminal B/Terminal C responds to Terminal A in the same process as intra-subnet known unicast packet forwarding.

Inter-Subnet Packet Forwarding

Inter-subnet packets must be forwarded through a Layer 3 gateway. Figure 1-1036 shows inter-subnet packet forwarding in centralized VXLAN gateway scenarios.

Figure 1-1036 Inter-subnet packet forwarding

1. After Leaf 1 receives Host 1's packet, it determines the Layer 2 BD of the packet based on the access interface and VLAN information and searches for the outbound interface and encapsulation information in the BD.
2. Leaf 1's VTEP performs VXLAN encapsulation based on the outbound interface and encapsulation information and forwards the packets to Spine.
3. After Spine receives the VXLAN packet, it decapsulates the packet and finds that the destination MAC address of the inner packet is the MAC address (MAC3) of the Layer 3 gateway interface (VBDIF10) so that the packet must be forwarded at Layer 3.
4. Spine removes the inner Ethernet header, parses the destination IP address, and searches the routing table for a next hop address. Spine then searches the ARP table based on the next hop address to obtain the destination MAC address, VXLAN tunnel's outbound interface, and VNI.
5. Spine performs VXLAN encapsulation on the inner packet again and forwards the VXLAN packet to Leaf 2, with the source MAC address in the inner Ethernet header being the MAC address (MAC4) of the Layer 3 gateway interface (VBDIF20).
6. Upon receipt of the VXLAN packet, Leaf 2's VTEP verifies the VXLAN packet based on the UDP destination port number, source and destination IP addresses, and VNI. Leaf 2 then obtains the Layer 2 broadcast domain based on the VNI and removes the outer headers to obtain the inner Layer 2 packet. It then searches for the outbound interface and encapsulation information in the Layer 2 broadcast domain.
7. Leaf 2 adds VLAN tags to the packets based on the outbound interface and encapsulation information and forwards the packets to Host 2.

Host 2 sends packets to Host 1 in the same manner.

VXLAN Tunnel Establishment

A VXLAN tunnel is identified by a pair of VTEP IP addresses. During VXLAN tunnel establishment, the local and remote VTEPs attempt to obtain IP addresses of each other. A VXLAN tunnel can be established if the IP addresses obtained are routable at Layer 3. When BGP EVPN is used to dynamically establish a VXLAN tunnel, the local and remote VTEPs first establish a BGP EVPN peer relationship and then exchange BGP EVPN routes to transmit VNIs and VTEP IP addresses.

As shown in Figure 1-1037, two hosts connect to Leaf1, one host connects to Leaf2, and a Layer 3 gateway is deployed on the spine node. A VXLAN tunnel needs to be established between Leaf1 and Leaf2 to implement communication between Host3 and Host2. To implement communication between Host1 and Host2, a VXLAN tunnel needs to be established between Leaf1 and Spine and between Spine and Leaf2. Though Host1 and Host3 both connect to Leaf1, they belong to different subnets and need to communicate through the Layer 3 gateway deployed on Spine. Therefore, a VXLAN tunnel needs to be created between Leaf1 and Spine.

A VXLAN tunnel is determined by a pair of VTEP IP addresses. When a local VTEP receives the same remote VTEP IP address repeatedly, only one VXLAN tunnel can be established, but packets are encapsulated with different VNIs before being forwarded through the tunnel.

Figure 1-1037 VXLAN tunnel networking

The following example illustrates how to dynamically establish a VXLAN tunnel using BGP EVPN between Leaf1 and Leaf2 on the network shown in Figure 1-1038.

Figure 1-1038 Dynamic VXLAN tunnel establishment

1. First, a BGP EVPN peer relationship is established between Leaf1 and Leaf2. Then, Layer 2 broadcast domains are created on Leaf1 and Leaf2, and VNIs are bound to the Layer 2 broadcast domains. Next, an EVPN instance is configured in each Layer 2 broadcast domain, and an RD, export VPN target (ERT), and import VPN target (IRT) are configured for the EVPN instance. After the local VTEP IP address is configured on Leaf1 and Leaf2, they generate a BGP EVPN route and send it to each other. The BGP EVPN route carries the local EVPN instance's ERT, Next_Hop attribute, and an inclusive multicast route (Type 3 route defined in BGP EVPN). Figure 1-1039 shows the format of an inclusive multicast route, which comprises a prefix and a PMSI attribute. VTEP IP addresses are stored in the Originating Router's IP Address field in the inclusive multicast route prefix, and VNIs are stored in the MPLS Label field in the PMSI attribute. The VTEP IP address is also included in the Next_Hop attribute.

   Figure 1-1039 Format of an inclusive multicast route
   

2. After Leaf1 and Leaf2 receive a BGP EVPN route from each other, they match the ERT of the route against the IRT of the local EVPN instance. If a match is found, the route is accepted. If no match is found, the route is discarded. Leaf1 and Leaf2 obtain the peer VTEP IP address (from the Next_Hop attribute) and VNI carried in the route. If the peer VTEP IP address is reachable at Layer 3, they establish a VXLAN tunnel to the peer end. Moreover, the local end creates a VNI-based ingress replication table and adds the peer VTEP IP address to the table for forwarding BUM packets.

The process of dynamically establishing VXLAN tunnels between Leaf1 and Spine and between Leaf2 and Spine using BGP EVPN is similar to the preceding process.

A VPN target is an extended community attribute of BGP. An EVPN instance can have the IRT and ERT configured. The local EVPN instance's ERT must match the remote EVPN instance's IRT for EVPN route advertisement. If not, VXLAN tunnels cannot be dynamically established. If only one end can successfully accept the BGP EVPN route, this end can establish a VXLAN tunnel to the other end, but cannot exchange data packets with the other end. The other end drops packets after confirming that there is no VXLAN tunnel to the end that has sent these packets.

For details about VPN targets, see Basic BGP/MPLS IP VPN.

Dynamic MAC Address Learning

VXLAN supports dynamic MAC address learning to allow communication between tenants. MAC address entries are dynamically created and do not need to be manually maintained, greatly reducing maintenance workload. The following example illustrates dynamic MAC address learning for intra-subnet communication of hosts on the network shown in Figure 1-1040.

Figure 1-1040 Dynamic MAC address learning

1. Host3 sends dynamic ARP packets when it first communicates with Leaf1. Leaf1 learns the MAC address of Host3 and the mapping between the BDID and packet inbound interface (that is, the physical interface Port 1 corresponding to the Layer 2 sub-interface), and generates a MAC address entry about Host3 in the local MAC address table, with the outbound interface being Port 1. Leaf1 generates a BGP EVPN route based on the ARP entry of Host3 and sends it to Leaf2. The BGP EVPN route carries the local EVPN instance's ERT, Next_Hop attribute, and a Type 2 route (MAC/IP route) defined in BGP EVPN. The Next_Hop attribute carries the local VTEP's IP address. The MAC Address Length and MAC Address fields identify Host3's MAC address. The Layer 2 VNI is stored in the MPLS Label1 field. Figure 1-1041 shows the format of a MAC route or an IP route.

   Figure 1-1041 Format of a MAC/IP route
   

2. After receiving the BGP EVPN route from Leaf1, Leaf2 matches the ERT of the EVPN instance carried in the route against the IRT of the local EVPN instance. If a match is found, the route is accepted. If no match is found, the route is discarded. After accepting the route, Leaf2 obtains the MAC address of Host3 and the mapping between the BDID and the VTEP IP address (Next_Hop attribute) of Leaf1, and generates the MAC address entry of the Host3 in the local MAC address table. The recursion to the outbound interface needs to be performed based on the next hop, and the final recursion result is the VXLAN tunnel destined for Leaf1.

Leaf1 learns the MAC route of Host2 in a similar process.

• When hosts on different subnets communicate with each other, only the hosts and Layer 3 gateway need to dynamically learn MAC addresses from each other. This process is similar to the preceding process.

• Leaf nodes can learn the MAC addresses of hosts during data forwarding, depending on their capabilities to learn MAC addresses from data packets. If VXLAN tunnels are established using BGP EVPN, leaf nodes can dynamically learn the MAC addresses of hosts through BGP EVPN routes, rather than during data forwarding.

Intra-subnet Forwarding of Known Unicast Packets

Intra-subnet known unicast packets are forwarded only between Layer 2 VXLAN gateways and are unknown to Layer 3 VXLAN gateways. Figure 1-1042 shows the forwarding process of known unicast packets.

Figure 1-1042 Intra-subnet forwarding of known unicast packets

1. After Leaf1 receives a packet from Host3, it determines the Layer 2 broadcast domain of the packet based on the access interface and VLAN information, and searches for the outbound interface and encapsulation information in the broadcast domain.
2. Leaf1's VTEP performs VXLAN encapsulation based on the obtained encapsulation information and forwards the packet through the outbound interface obtained.
3. After the VTEP on Leaf2 receives the VXLAN packet, it checks the UDP destination port number, source and destination IP addresses, and VNI of the packet to determine the packet validity. Leaf2 obtains the Layer 2 broadcast domain based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
4. Leaf2 obtains the destination MAC address of the inner Layer 2 packet, adds a VLAN tag to the packet based on the outbound interface and encapsulation information in the local MAC address table, and forwards the packet to Host2.

Host2 sends packets to Host3 in the same process.

Intra-subnet Forwarding of BUM Packets

Intra-subnet BUM packets are forwarded only between Layer 2 VXLAN gateways, and are unknown to Layer 3 VXLAN gateways. Intra-subnet BUM packets can be forwarded in ingress replication mode. In this mode, when a BUM packet enters a VXLAN tunnel, the access-side VTEP performs VXLAN encapsulation, and then forwards the packet to all egress VTEPs that are in the ingress replication list. When the BUM packet leaves the VXLAN tunnel, the egress VTEP decapsulates the packet. Figure 1-1043 shows the forwarding process of BUM packets.

Figure 1-1043 Intra-subnet forwarding of BUM packets in ingress replication mode

1. After Leaf1 receives a packet from TerminalA, it determines the Layer 2 broadcast domain of the packet based on the access interface and VLAN information in the packet.
2. Leaf1's VTEP obtains the ingress replication list for the VNI, replicates the packet based on the list, and performs VXLAN encapsulation. Leaf1 then forwards the VXLAN packet through the outbound interface.
3. After the VTEP on Leaf2 or Leaf3 receives the VXLAN packet, it checks the UDP destination port number, source and destination IP addresses, and VNI of the packet to determine the packet validity. Leaf2 or Leaf3 obtains the Layer 2 broadcast domain based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
4. Leaf2 or Leaf3 checks the destination MAC address of the inner Layer 2 packet and finds it a BUM MAC address. Therefore, Leaf2 or Leaf3 broadcasts the packet onto the network connected to terminals (not the VXLAN tunnel side) in the Layer 2 broadcast domain. Specifically, Leaf2 or Leaf3 finds the outbound interfaces and encapsulation information not related to the VXLAN tunnel, adds VLAN tags to the packet, and forwards the packet to TerminalB or TerminalC.

The forwarding process of a response packet from TerminalB/TerminalC to TerminalA is similar to the intra-subnet forwarding process of known unicast packets.

Inter-subnet Packet Forwarding

Inter-subnet packets must be forwarded through a Layer 3 gateway. Figure 1-1044 shows the inter-subnet packet forwarding process in centralized VXLAN gateway scenarios.

Figure 1-1044 Inter-subnet packet forwarding

1. After Leaf1 receives a packet from Host1, it determines the Layer 2 broadcast domain of the packet based on the access interface and VLAN in the packet, and searches for the outbound interface and encapsulation information in the Layer 2 broadcast domain.
2. The VTEP on Leaf1 performs VXLAN tunnel encapsulation based on the outbound interface and encapsulation information, and forwards the packet to Spine.
3. Spine decapsulates the received VXLAN packet, finds that the destination MAC address in the inner packet is MAC3 of the Layer 3 gateway interface VBDIF10, and determines that the packet needs to be forwarded at Layer 3.
4. Spine removes the Ethernet header of the inner packet and parses the destination IP address. It then searches the routing table based on the destination IP address to obtain the next hop address, and searches ARP entries based on the next hop to obtain the destination MAC address, VXLAN tunnel outbound interface, and VNI.
5. Spine re-encapsulates the VXLAN packet and forwards it to Leaf2. The source MAC address in the Ethernet header of the inner packet is MAC4 of the Layer 3 gateway interface VBDIF20.
6. After the VTEP on Leaf2 receives the VXLAN packet, it checks the UDP destination port number, source and destination IP addresses, and VNI of the packet to determine the packet validity. The VTEP then obtains the Layer 2 broadcast domain based on the VNI, decapsulates the packet to obtain the inner Layer 2 packet, and searches for the outbound interface and encapsulation information in the corresponding Layer 2 broadcast domain.
7. Leaf2 adds a VLAN tag to the packet based on the outbound interface and encapsulation information, and forwards the packet to Host2.

Host2 sends packets to Host1 through a similar process.

VXLAN Tunnel Establishment

A VXLAN tunnel is identified by a pair of VTEP IP addresses. During VXLAN tunnel establishment, the local and remote VTEPs attempt to obtain IP addresses of each other. A VXLAN tunnel can be established if the IP addresses obtained are routable at Layer 3. When BGP EVPN is used to dynamically establish a VXLAN tunnel, the local and remote VTEPs first establish a BGP EVPN peer relationship and then exchange BGP EVPN routes to transmit VNIs and VTEP IP addresses.

In distributed VXLAN gateway scenarios, leaf nodes function as both Layer 2 and Layer 3 VXLAN gateways. Spine nodes are unaware of the VXLAN tunnels and only forward VXLAN packets between different leaf nodes. On the control plane, a VXLAN tunnel only needs to be set up between leaf nodes. In Figure 1-1045, a VXLAN tunnel is established between Leaf1 and Leaf2 for Host1 and Host2 or Host3 and Host2 to communicate. Because Host1 and Host3 both connect to Leaf1, they can directly communicate through Leaf1 instead of over a VXLAN tunnel.

A VXLAN tunnel is determined by a pair of VTEP IP addresses. When a local VTEP receives the same remote VTEP IP address repeatedly, only one VXLAN tunnel can be established, but packets are encapsulated with different VNIs before being forwarded through the tunnel.

Figure 1-1045 VXLAN tunnel networking

In distributed gateway scenarios, BGP EVPN can be used to dynamically establish VXLAN tunnels in either of the following situations:

Intra-subnet Communication

On the network shown in Figure 1-1046, intra-subnet communication between Host2 and Host3 requires only Layer 2 forwarding. The process for establishing a VXLAN tunnel using BGP EVPN is as follows.

Figure 1-1046 Dynamic VXLAN tunnel establishment (1)

1. First, a BGP EVPN peer relationship is established between Leaf1 and Leaf2. Then, Layer 2 broadcast domains are created on Leaf1 and Leaf2, and VNIs are bound to the Layer 2 broadcast domains. Next, an EVPN instance is configured in each Layer 2 broadcast domain, and an RD, an ERT, and an IRT are configured for the EVPN instance. After the local VTEP IP address is configured on Leaf1 and Leaf2, they generate a BGP EVPN route and send it to each other. The BGP EVPN route carries the local EVPN instance's ERT and an inclusive multicast route (Type 3 route defined in BGP EVPN). Figure 1-1047 shows the format of an inclusive multicast route, which comprises a prefix and a PMSI attribute. VTEP IP addresses are stored in the Originating Router's IP Address field in the inclusive multicast route prefix, and VNIs are stored in the MPLS Label field in the PMSI attribute. The VTEP IP address is also included in the Next_Hop attribute.

   Figure 1-1047 Format of an inclusive multicast route
   

2. After Leaf1 and Leaf2 receive a BGP EVPN route from each other, they match the ERT of the route against the IRT of the local EVPN instance. If a match is found, the route is accepted. If no match is found, the route is discarded. Leaf1 and Leaf2 obtain the peer VTEP IP address (from the Next_Hop attribute) and VNI carried in the route. If the peer VTEP IP address is reachable at Layer 3, they establish a VXLAN tunnel to the peer end. Moreover, the local end creates a VNI-based ingress replication table and adds the peer VTEP IP address to the table for forwarding BUM packets.

A VPN target is an extended community attribute of BGP. An EVPN instance can have the IRT and ERT configured. The local EVPN instance's ERT must match the remote EVPN instance's IRT for EVPN route advertisement. If not, VXLAN tunnels cannot be dynamically established. If only one end can successfully accept the BGP EVPN route, this end can establish a VXLAN tunnel to the other end, but cannot exchange data packets with the other end. The other end drops packets after confirming that there is no VXLAN tunnel to the end that has sent these packets.

For details about VPN targets, see Basic BGP/MPLS IP VPN.

Inter-Subnet Communication

Inter-subnet communication between Host1 and Host2 requires Layer 3 forwarding. When VXLAN tunnels are established using BGP EVPN, Leaf1 and Leaf2 must advertise host IP routes. Typically, 32-bit host IP routes are advertised. Because different leaf nodes may connect to the same network segment on the VXLAN network, the network segment routes advertised by the leaf nodes may conflict. This conflict may cause host unreachability of some leaf nodes. Leaf nodes can advertise network segment routes in the following scenarios:

• The network segment that a leaf node connects to is unique on a VXLAN, and a large number of specific host routes are available. In this case, the routes of the network segment to which the host IP routes belong can be advertised so that leaf nodes do not have to store all these routes.

• When hosts on a VXLAN need to access external networks, leaf nodes can advertise routes destined for external networks onto the VXLAN to allow other leaf nodes to learn the routes.

Before establishing a VXLAN tunnel, perform configurations listed in the following table on Leaf1 and Leaf2.

Dynamic VXLAN tunnel establishment varies depending on how host IP routes are advertised.

• Host IP routes are advertised through IRB routes. (Figure 1-1048 shows the process.)

  Figure 1-1048 Dynamic VXLAN tunnel establishment (2)
  

  1. When Host1 communicates with Leaf1 for the first time, Leaf1 learns the ARP entry of Host1 after receiving dynamic ARP packets. Leaf1 then finds the L3VPN instance bound to the VBDIF interface of the Layer 2 broadcast domain where Host1 resides, and obtains the Layer 3 VNI associated with the L3VPN instance. The EVPN instance of Leaf1 then generates an IRB route based on the information obtained. Figure 1-1049 shows the IRB route. The host IP address is stored in the IP Address Length and IP Address fields; the Layer 3 VNI is stored in the MPLS Label2 field.

     Figure 1-1049 IRB route
     

  2. Leaf1 generates and sends a BGP EVPN route to Leaf2. The BGP EVPN route carries the local EVPN instance's ERT, extended community attribute, Next_Hop attribute, and the IRB route. The extended community attribute carries the tunnel type (VXLAN tunnel) and local VTEP MAC address; the Next_Hop attribute carries the local VTEP IP address.

  3. After Leaf2 receives the BGP EVPN route from Leaf1, Leaf2 processes the route as follows:

     • If the ERT carried in the route is the same as the IRT of the local EVPN instance, the route is accepted. After the EVPN instance obtains IRB routes, it can extract ARP routes from the IRB routes for the advertisement of host ARP entries.

     • If the ERT carried in the route is the same as the eIRT of the local L3VPN instance, the route is accepted. Then, the L3VPN instance obtains the IRB route carried in the route, extracts the host IP address and Layer 3 VNI of Host1, and saves the host IP route of Host1 to the routing table. The outbound interface is obtained through recursion based on the next hop of the route. The final recursion result is the VXLAN tunnel to Leaf1, as shown in Figure 1-1050.

       A BGP EVPN route is discarded only when the ERT in the route is different from the local EVPN instance's IRT and local L3VPN instance's eIRT.

       Figure 1-1050 Remote host IP route information
       

     • If the route is accepted by the EVPN instance or L3VPN instance, Leaf2 obtains Leaf1's VTEP IP address from the Next_Hop attribute. If the VTEP IP address is routable at Layer 3, a VXLAN tunnel to Leaf1 is established.

  Leaf1 establishes a VXLAN tunnel to Leaf2 through a similar process.

• Host IP routes are advertised through IP prefix routes, as shown in Figure 1-1051.

  Figure 1-1051 Dynamic VXLAN tunnel establishment (3)
  

  1. Leaf1 generates a direct route to Host1's IP address. Then, Leaf1 has an L3VPN instance configured to import the direct route, so that Host1's IP route is saved to the routing table of the L3VPN instance and the Layer 3 VNI associated with the L3VPN instance is added. Figure 1-1052 shows the local host IP route.

     Figure 1-1052 Local host IP route information
     

     If network segment route advertisement is required, use a dynamic routing protocol, such as OSPF. Then configure an L3VPN instance to import the routes of the dynamic routing protocol.

  2. Leaf1 is configured to advertise IP prefix routes in the L3VPN instance. Figure 1-1053 shows the IP prefix route. The host IP address is stored in the IP Prefix Length and IP Prefix fields; the Layer 3 VNI is stored in the MPLS Label field. Leaf1 generates and sends a BGP EVPN route to Leaf2. The BGP EVPN route carries the local L3VPN instance's eERT, extended community attribute, Next_Hop attribute, and the IP prefix route. The extended community attribute carries the tunnel type (VXLAN tunnel) and local VTEP MAC address; the Next_Hop attribute carries the local VTEP IP address.

     Figure 1-1053 Format of an IP prefix route
     

  3. After Leaf2 receives the BGP EVPN route from Leaf1, Leaf2 processes the route as follows:

     • Matches the eERT of the route against the eIRT of the local L3VPN instance. If a match is found, the route is accepted. Then, the L3VPN instance obtains the IP prefix type route carried in the route, extracts the host IP address and Layer 3 VNI of Host1, and saves the host IP route of Host1 to the routing table. The outbound interface is obtained through recursion based on the next hop of the route. The final recursion result is the VXLAN tunnel to Leaf1, as shown in Figure 1-1054.

       Figure 1-1054 Remote host IP route information
       

     • If the route is accepted by the EVPN instance or L3VPN instance, Leaf2 obtains Leaf1's VTEP IP address from the Next_Hop attribute. If the VTEP IP address is routable at Layer 3, a VXLAN tunnel to Leaf1 is established.

  Leaf1 establishes a VXLAN tunnel to Leaf2 through a similar process.

Dynamic MAC address learning

VXLAN supports dynamic MAC address learning to allow communication between tenants. MAC address entries are dynamically created and do not need to be manually maintained, greatly reducing maintenance workload. In distributed VXLAN gateway scenarios, inter-subnet communication requires Layer 3 forwarding; MAC address learning is implemented using dynamic ARP packets between the local host and gateway. The following example illustrates dynamic MAC address learning for intra-subnet communication of hosts on the network shown in Figure 1-1055.

Figure 1-1055 Dynamic MAC address learning

1. Host3 sends dynamic ARP packets when it first communicates with Leaf1. Leaf1 learns the MAC address of Host3 and the mapping between the BDID and packet inbound interface (that is, the physical interface Port 1 corresponding to the Layer 2 sub-interface), and generates a MAC address entry about Host3 in the local MAC address table, with the outbound interface being Port 1. Leaf1 generates a BGP EVPN route based on the ARP entry of Host3 and sends it to Leaf2. The BGP EVPN route carries the local EVPN instance's ERT, Next_Hop attribute, and a Type 2 route (MAC/IP route) defined in BGP EVPN. The Next_Hop attribute carries the local VTEP's IP address. The MAC Address Length and MAC Address fields identify Host3's MAC address. The Layer 2 VNI is stored in the MPLS Label1 field. Figure 1-1056 shows the format of a MAC route or an IP route.

   Figure 1-1056 Format of a MAC/IP route
   

2. After receiving the BGP EVPN route from Leaf1, Leaf2 matches the ERT of the EVPN instance carried in the route against the IRT of the local EVPN instance. If a match is found, the route is accepted. If no match is found, the route is discarded. After accepting the route, Leaf2 obtains the MAC address of Host3 and the mapping between the BDID and the VTEP IP address (Next_Hop attribute) of Leaf1, and generates the MAC address entry of the Host3 in the local MAC address table. The outbound interface is obtained through recursion based on the next hop, and the final recursion result is the VXLAN tunnel destined for Leaf1.

Leaf1 learns the MAC route of Host2 through a similar process.

Leaf nodes can learn the MAC addresses of hosts during data forwarding, depending on their capabilities to learn MAC addresses from data packets. If VXLAN tunnels are established using BGP EVPN, leaf nodes can dynamically learn the MAC addresses of hosts through BGP EVPN routes, rather than during data forwarding.

Intra-subnet Forwarding of Known Unicast Packets

Intra-subnet known unicast packets are forwarded only between Layer 2 VXLAN gateways and are unknown to Layer 3 VXLAN gateways. Figure 1-1057 shows the forwarding process of known unicast packets.

Figure 1-1057 Intra-subnet forwarding of known unicast packets

1. After Leaf1 receives a packet from Host3, it determines the Layer 2 broadcast domain of the packet based on the access interface and VLAN information, and searches for the outbound interface and encapsulation information in the broadcast domain.
2. Leaf1's VTEP performs VXLAN encapsulation based on the obtained encapsulation information and forwards the packet through the outbound interface obtained.
3. After the VTEP on Leaf2 receives the VXLAN packet, it checks the UDP destination port number, source and destination IP addresses, and VNI of the packet to determine the packet validity. Leaf2 obtains the Layer 2 broadcast domain based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
4. Leaf2 obtains the destination MAC address of the inner Layer 2 packet, adds a VLAN tag to the packet based on the outbound interface and encapsulation information in the local MAC address table, and forwards the packet to Host2.

Host2 sends packets to Host3 through a similar process.

Intra-subnet Forwarding of BUM Packets

Intra-subnet BUM packets are forwarded only between Layer 2 VXLAN gateways, and are unknown to Layer 3 VXLAN gateways. Intra-subnet BUM packets can be forwarded in ingress replication mode. In this mode, when a BUM packet enters a VXLAN tunnel, the access-side VTEP performs VXLAN encapsulation, and then forwards the packet to all egress VTEPs that are in the ingress replication list. When the BUM packet leaves the VXLAN tunnel, the egress VTEP decapsulates the packet. Figure 1-1058 shows the forwarding process of BUM packets.

Figure 1-1058 Intra-subnet forwarding of BUM packets in ingress replication mode

1. After Leaf1 receives a packet from TerminalA, it determines the Layer 2 broadcast domain of the packet based on the access interface and VLAN information in the packet.
2. Leaf1's VTEP obtains the ingress replication list for the VNI, replicates the packet based on the list, and performs VXLAN encapsulation. Leaf1 then forwards the VXLAN packet through the outbound interface.
3. After the VTEP on Leaf2 or Leaf3 receives the VXLAN packet, it checks the UDP destination port number, source and destination IP addresses, and VNI of the packet to determine the packet validity. Leaf2 or Leaf3 obtains the Layer 2 broadcast domain based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet.
4. Leaf2 or Leaf3 checks the destination MAC address of the inner Layer 2 packet and finds it a BUM MAC address. Therefore, Leaf2 or Leaf3 broadcasts the packet onto the network connected to terminals (not the VXLAN tunnel side) in the Layer 2 broadcast domain. Specifically, Leaf2 or Leaf3 finds the outbound interfaces and encapsulation information not related to the VXLAN tunnel, adds VLAN tags to the packet, and forwards the packet to TerminalB or TerminalC.

The forwarding process of a response packet from TerminalB/TerminalC to TerminalA is similar to the intra-subnet forwarding process of known unicast packets.

Inter-subnet Packet Forwarding

Inter-subnet packets must be forwarded through a Layer 3 gateway. Figure 1-1059 shows the inter-subnet packet forwarding process in distributed VXLAN gateway scenarios.

Figure 1-1059 Inter-subnet packet forwarding

1. After Leaf1 receives a packet from Host1, it finds that the destination MAC address of the packet is a gateway MAC address so that the packet must be forwarded at Layer 3.
2. Leaf1 first determines the Layer 2 broadcast domain of the packet based on the inbound interface and then finds the L3VPN instance to which the VBDIF interface of the Layer 2 broadcast domain is bound. Leaf1 searches the routing table of the L3VPN instance for a matching host route based on the destination IP address of the packet and obtains the Layer 3 VNI and next hop address corresponding to the route. Figure 1-1060 shows the host route in the L3VPN routing table. If the outbound interface is a VXLAN tunnel, Leaf1 determines that VXLAN encapsulation is required and then:
   • Obtains MAC addresses based on the VXLAN tunnel's source and destination IP addresses and replaces the source and destination MAC addresses in the inner Ethernet header.
   • Encapsulates the Layer 3 VNI into the packet.
   • Encapsulates the VXLAN tunnel's destination and source IP addresses in the outer header. The source MAC address is the MAC address of the outbound interface on Leaf1, and the destination MAC address is the MAC address of the next hop.
   Figure 1-1060 Host route information in the L3VPN routing table
   
3. The VXLAN packet is then transmitted over the IP network based on the IP and MAC addresses in the outer headers and finally reaches Leaf2.
4. After Leaf2 receives the VXLAN packet, it decapsulates the packet and finds that the destination MAC address is its own MAC address. It then determines that the packet must be forwarded at Layer 3.
5. Leaf2 finds the corresponding L3VPN instance based on the Layer 3 VNI carried in the packet. Then, Leaf2 searches the routing table of the L3VPN instance and finds that the next hop of the packet is the gateway interface address. Leaf2 then replaces the destination MAC address with the MAC address of Host2, replaces the source MAC address with the MAC address of Leaf2, and forwards the packet to Host2.

Host2 sends packets to Host1 in a similar process.

When Huawei devices need to communicate with non-Huawei devices, ensure that the non-Huawei devices use the same forwarding mode. Otherwise, the Huawei devices may fail to communicate with non-Huawei devices.

Background

To meet the requirements of inter-regional operations, user access, geographical redundancy, and other scenarios, an increasing number of enterprises deploy DCs across regions. Data Center Interconnect (DCI) is a solution that enables communication between VMs in different DCs. Using technologies such as VXLAN and BGP EVPN, DCI securely and reliably transmits DC packets over carrier networks. Three-segment VXLAN can be configured to enable inter-subnet communication between VMs in different DCs.

Benefits

Three-segment VXLAN enables Layer 3 communication between DC and offers the following benefits to users:

• Hosts in different DCs can communicate at Layer 3.
• Different DCs do not need to run the same routing protocol for communication.
• Different DCs do not require information orchestration for communication.

Implementation

Three-segment VXLAN establishes one VXLAN tunnel segment in each of the DCs and also establishes one VXLAN tunnel segment between the DCs. As shown in Figure 1-1061, BGP EVPN is used to create VXLAN tunnels in distributed gateway mode within both DC A and DC B so that the VMs in each DC can communicate with each other. Leaf2 and Leaf3 are the edge devices within the DCs that connect to the backbone network. BGP EVPN is used to configure a VXLAN tunnel between Leaf2 and Leaf3, so that the VXLAN packets received by one DC can be decapsulated, re-encapsulated, and sent to the peer DC. This process provides E2E transport for inter-DC VXLAN packets and ensures that VMs in different DCs can communicate with each other.

This function applies only to IPv4 over IPv4 networks.

In three-segment VXLAN scenarios, only VXLAN tunnels in distributed gateway mode can be deployed in DCs.

Figure 1-1061 Using three-segment VXLAN for DCI

Control Plane

The following describes how three-segment VXLAN tunnels are established.

The process of advertising routes on Leaf1 and Leaf4 is not described in this section. For details, see VXLAN Tunnel Establishment.

Data Packet Forwarding

A general overview of the packet forwarding process on Leaf1 and Leaf4 is provided below. For detailed information, see Intra-subnet Packet Forwarding.

Other Functions

Local leaking of EVPN routes is needed in scenarios where different VPN instances are used for the access of different services in a DC and but an external VPN instance is used to communicate with other DCs to block VPN instance allocation information within the DC from the outside. Depending on route sources, this function can be used in the following scenarios:

Local VPN routes are advertised through EVPN after being locally leaked

As shown in Figure 1-1064, the process is as follows:

1. The function to import VPN routes to a local VPN instance named vpn1 is configured in the BGP VPN instance IPv4 or IPv6 address family.
2. vpn1 sends received routes to the VPNv4 or VPNv6 component, which then checks whether the ERT of vpn1 is the same as the IRT of the external VPN instance vpn2. If they are the same, the VPNv4 or VPNv6 component imports these routes to vpn2.
3. vpn2 sends locally leaked routes to the EVPN component and advertises these routes as BGP EVPN routes to peers. In this case, vpn2 must be able to advertise locally leaked routes as BGP EVPN routes.

Figure 1-1064 Local leaking of EVPN routes (1)

Remote public network routes are advertised through EVPN after being locally leaked

As shown in Figure 1-1065, the process is as follows:

1. The EVPN component receives public network routes from a remote peer.
2. The EVPN component imports the received routes to vpn1.
3. vpn1 sends received routes to the VPNv4 or VPNv6 component, which then checks whether the ERT of vpn1 is the same as the IRT of vpn2. If they are the same, the VPNv4 or VPNv6 component imports these routes to vpn2. In this case, vpn1 must be able to perform remote and local route leaking route leaking in succession.
4. vpn2 sends locally leaked routes to the EVPN component and advertises these routes as BGP EVPN routes to peers. In this case, vpn2 must be able to advertise locally leaked routes as BGP EVPN routes.

Figure 1-1065 Local leaking of EVPN routes (2)

Background

Figure 1-1066 shows the scenario where three-segment VXLAN is deployed to implement Layer 2 interconnection between DCs. VXLAN tunnels are configured both within DC A and DC B and between transit leaf nodes in both DCs. To enable communication between VM1 and VM2, implement Layer 2 communication between DC A and DC B. If the VXLAN tunnels within DC A and DC B use the same VXLAN Network Identifier (VNI), this VNI can also be used to establish a VXLAN tunnel between Transit Leaf1 and Transit Leaf2. In practice, however, different DCs have their own VNI spaces. Therefore, the VXLAN tunnels within DC A and DC B tend to use different VNIs. In this case, to establish a VXLAN tunnel between Transit Leaf1 and Transit Leaf2, VNIs conversion must be implemented.

Figure 1-1066 Deployment of three-segment VXLAN for Layer 2 interworking

Benefits

This solution offers the following benefits to users:

• Implements Layer 2 interconnection between hosts in different DCs.

• Decouples the VNI space of the network within a DC from that of the network between DCs, simplifying network maintenance.

• Isolates network faults within a DC from those between DCs, facilitating fault location.

Principles

Currently, this solution is implemented in the local VNI mode. It is similar to downstream label allocation. The local VNI of the peer transit leaf node functions as the outbound VNI, which is used by packets that the local transit leaf node sends to the peer transit leaf node for VXLAN encapsulation.

Control Plane

This function is only supported for IPv4 over IPv4 networks.

The establishment of VXLAN tunnels between leaf nodes is the same as VXLAN tunnel establishment for intra-subnet interworking in common VXLAN scenarios. Therefore, the detailed process is not described here. Regarding the control plane, MAC address learning by a host is described here.

On the network shown in Figure 1-1067, the control plane is implemented as follows:

Figure 1-1067 Control plane for VXLAN mapping in local VNI mode

1. Server Leaf1 learns VM1's MAC address, generates a BGP EVPN route, and sends it to Transit Leaf1. The BGP EVPN route contains the following information:

   • Type 2 route: EVPN instance's RD value, VM1's MAC address, and Server Leaf1's local VNI.

   • Next hop: Server Leaf1's VTEP IP address.

   • Extended community attribute: encapsulated tunnel type (VXLAN).

   • ERT: EVPN instance's export RT value.

2. Upon receipt, Transit Leaf1 adds the BGP EVPN route to its local EVPN instance and generates a MAC address entry for VM1 in the EVPN instance-bound BD. Based on the next hop and encapsulated tunnel type, the MAC address entry's outbound interface recurses to the VXLAN tunnel destined for Server Leaf1. The VNI in VXLAN tunnel encapsulation information is Transit Leaf1's local VNI.

3. Transit Leaf1 re-originates the BGP EVPN route and then advertises the route to Transit Leaf2. The re-originated BGP EVPN route contains the following information:

   • Type 2 route: EVPN instance's RD value, VM1's MAC address, and Transit Leaf1's local VNI.

   • Next hop: Transit Leaf1's VTEP IP address.

   • Extended community attribute: encapsulated tunnel type (VXLAN).

   • ERT: EVPN instance's export RT value.

4. Upon receipt, Transit Leaf2 adds the re-originated BGP EVPN route to its local EVPN instance and generates a MAC address entry for VM1 in the EVPN instance-bound BD. Based on the next hop and encapsulated tunnel type, the MAC address entry's outbound interface recurses to the VXLAN tunnel destined for Transit Leaf1. The outbound VNI in VXLAN tunnel encapsulation information is Transit Leaf1's local VNI.

5. Transit Leaf2 re-originates the BGP EVPN route and then advertises the route to Server Leaf2. The re-originated BGP EVPN route contains the following information:

   • Type 2 route: EVPN instance's RD value, VM1's MAC address, and Transit Leaf2's local VNI.

   • Next hop: Transit Leaf2's VTEP IP address.

   • Extended community attribute: encapsulated tunnel type (VXLAN).

   • ERT: EVPN instance's export RT value.

6. Upon receipt, Server Leaf2 adds the re-originated BGP EVPN route to its local EVPN instance and generates a MAC address entry for VM1 in the EVPN instance-bound BD. Based on the next hop and encapsulated tunnel type, the MAC address entry's outbound interface recurses to the VXLAN tunnel destined for Transit Leaf2. The VNI in VXLAN tunnel encapsulation information is Server Leaf2's local VNI.

The preceding process takes MAC address learning by VM1 for example. MAC address learning by VM2 is the same, which is not described here.

Forwarding Plane

Figure 1-1068 shows the known unicast packets are forwarded. The following example process shows how VM2 sends Layer 2 packets to VM1:

Figure 1-1068 Known unicast packet forwarding with VXLAN mapping in local VNI mode

1. After receiving a Layer 2 packet from VM2 through a BD Layer 2 sub-interface, Server Leaf2 searches the BD's MAC address table based on the destination MAC address for the VXLAN tunnel's outbound interface and obtains VXLAN tunnel encapsulation information (local VNI, destination VTEP IP address, and source VTEP IP address). Based on the obtained information, the Layer 2 packet is encapsulated through the VXLAN tunnel and then forwarded to Transit Leaf2.

2. Upon receipt, Transit Leaf2 decapsulates the VXLAN packet, finds the target BD based on the VNI, searches the BD's MAC address table based on the destination MAC address for the VXLAN tunnel's outbound interface, and obtains the VXLAN tunnel encapsulation information (outbound VNI, destination VTEP IP address, and source VTEP IP address). Based on the obtained information, the Layer 2 packet is encapsulated through the VXLAN tunnel and then forwarded to Transit Leaf1.

3. Upon receipt, Transit Leaf1 decapsulates the VXLAN packet. Because the packet's VNI is Transit Leaf1's local VNI, the target BD can be found based on this VNI. Transit Leaf1 also searches the BD's MAC address table based on the destination MAC address for the VXLAN tunnel's outbound interface and obtains the VXLAN tunnel encapsulation information (local VNI, destination VTEP IP address, and source VTEP IP address). Based on the obtained information, the Layer 2 packet is encapsulated through the VXLAN tunnel and then forwarded to Server Leaf1.

4. Upon receipt, Server Leaf1 decapsulates the VXLAN packet and forwards it at Layer 2 to VM1.

In the scenario with three-segment VXLAN for Layer 2 interworking, BUM packet forwarding is the same as that in the common VXLAN scenario except that the split horizon group is used to prevent loops. The similarities are not described here.

• After receiving BUM packets from a Server Leaf node in the same DC, a Transit Leaf node obtains the split horizon group to which the source VTEP belongs. Because all nodes in the same DC belong to the default split horizon group, BUM packets will not be replicated to other Server Leaf nodes within the DC. Because the peer Transit Leaf node belongs to a different split horizon group, BUM packets will be replicated to the peer Transit Leaf node.

• Upon receipt, the peer Transit Leaf node obtains the split horizon group to which the source VTEP belongs. Because the Transit Leaf nodes at both ends belong to the same split horizon group, BUM packets will not be replicated to the peer Transit Leaf node. Because the Server Leaf nodes within the DC belong to a different split horizon group, BUM packets will be replicated to them.

Basic Concepts

The network in Figure 1-1069 shows a scenario where an enterprise site (CPE) connects to a data center. The VPN GWs (PE1 and PE2) and CPE are connected through VXLAN tunnels to exchange the L2/L3 services between the CPE and data center. The data center gateway (CE1) is dual-homed to PE1 and PE2 to access the VXLAN network for enhanced network access reliability. If one PE fails, services can be rapidly switched to the other PE, minimizing service loss.

PE1 and PE2 on the network use the same virtual address as an NVE interface address (Anycast VTEP address) at the network side. In this way, the CPE is aware of only one remote NVE interface. After the CPE establishes a VXLAN tunnel with this virtual address, the packets from the CPE can reach CE1 through either PE1 or PE2. However, when a single-homed CE, such as CE2 or CE3, exists on the network, the packets from the CPE to the single-homed CE may need to detour to the other PE after reaching one PE. To achieve PE1-PE2 reachability, a bypass VXLAN tunnel needs to be established between PE1 and PE2. To establish this tunnel, an EVPN peer relationship is established between PE1 and PE2, and different addresses, namely, bypass VTEP addresses, are configured for PE1 and PE2.

Figure 1-1069 Basic networking of the VXLAN active-active scenario

Control Plane

• PE1 and PE2 exchange Inclusive Multicast routes (Type 3) whose source IP address is their shared anycast VTEP address. Each route carries a bypass VXLAN extended community attribute, which contains the bypass VTEP address of PE1 or PE2.

• After receiving the Inclusive Multicast route from each other, PE1 and PE2 consider that they form an anycast relationship based on the following details: The source IP address (anycast VTEP address) of the route is identical to PE1's and PE2's local virtual addresses, and the route carries a bypass VXLAN extended community attribute. PE1 and PE2 then establish a bypass VXLAN tunnel between them.

• PE1 and PE2 learn the MAC addresses of the CEs through the upstream packets from the AC side and advertise the MAC/IP routes (Type 2) to each other. The routes carry the ESIs of the access links of the CEs, information about the VLANs that the CEs access, and the bypass VXLAN extended community attribute.

• PE1 and PE2 learn the MAC address of the CPE through downstream packets from the network side. After learning that the next-hop address of the MAC route can be recursed to a static VXLAN tunnel, PE1 and PE2 advertise the route to each other through an MAC/IP route, without changing the next-hop address.

Data Packets Processing

• Layer 2 unicast packet forwarding

  • Uplink

    As shown in Figure 1-1070, after receiving Layer 2 unicast packets destined for the CPE from CE1, CE2, and CE3, PE1 and PE2 search for their local MAC address table to obtain outbound interfaces, perform VXLAN encapsulation on the packets, and forward them to the CPE.

    Figure 1-1070 Uplink unicast packet forwarding
    

  • Downlink

    As shown in Figure 1-1071:

    After receiving a Layer 2 unicast packet sent by the CPE to CE1, PE1 performs VXLAN decapsulation on the packet, searches the local MAC address table for the destination MAC address, obtains the outbound interface, and forwards the packet to CE1.

    After receiving a Layer 2 unicast packet sent by the CPE to CE2, PE1 performs VXLAN decapsulation on the packet, searches the local MAC address table for the destination MAC address, obtains the outbound interface, and forwards the packet to CE2.

    After receiving a Layer 2 unicast packet sent by the CPE to CE3, PE1 performs VXLAN decapsulation on the packet, searches the local MAC address table for the destination MAC address, and forwards it to PE2 over the bypass VXLAN tunnel. After the packet reaches PE2, PE2 searches the destination MAC address, obtains the outbound interface, and forwards the packet to CE3.

    The process for PE2 to forward packets from the CPE is the same as that for PE1 to forward packets from the CPE.

    Figure 1-1071 Downlink unicast packet forwarding
    

• BUM packet forwarding

  • As shown in Figure 1-1072, if the destination address of a BUM packet from the CPE is the Anycast VTEP address of PE1 and PE2, the BUM packet may be forwarded to either PE1 or PE2. If the BUM packet reaches PE2 first, PE2 sends a copy of the packet to CE3 and CE1. In addition, PE2 sends a copy of the packet to PE1 through the bypass VXLAN tunnel between PE1 and PE2. After the copy of the packet reaches PE1, PE1 sends it to CE2, not to the CPE or CE1. In this way, CE1 receives only one copy of the packet.

    Figure 1-1072 BUM packets from the CPE
    

  • As shown in Figure 1-1073, after a BUM packet from CE2 reaches PE1, PE1 sends a copy of the packet to CE1 and the CPE. In addition, PE1 sends a copy of the packet to PE2 through the bypass VXLAN tunnel between PE1 and PE2. After the copy of the packet reaches PE2, PE2 sends it to CE3, not to the CPE or CE1.

    Figure 1-1073 BUM packets from CE2
    

  • As shown in Figure 1-1074, after a BUM packet from CE1 reaches PE1, PE1 sends a copy of the packet to CE2 and the CPE. In addition, PE1 sends a copy of the packet to PE2 through the bypass VXLAN tunnel between PE1 and PE2. After the copy of the packet reaches PE2, PE2 sends it to CE3, not to the CPE or CE1.

    Figure 1-1074 BUM packets from CE1
    

• Layer 3 packets transmitted on the same subnet

  • Uplink

    As shown in Figure 1-1070, after receiving Layer 3 unicast packets destined for the CPE from CE1, CE2, and CE3, PE1 and PE2 search for the destination address and directly forward them to the CPE because they are on the same network segment.

  • Downlink

    As shown in Figure 1-1071:

    After the Layer 3 unicast packet sent from the CPE to CE1 reaches PE1, PE1 searches for the destination address and directly sends it to CE1 because they are on the same network segment.

    After the Layer 3 unicast packet sent from the CPE to CE2 reaches PE1, PE1 searches for the destination address and directly sends it to CE2 because they are on the same network segment.

    After the Layer 3 unicast packet sent from the CPE to CE3 reaches PE1, PE1 searches for the destination address and sends it to PE2, then sends it to CE3, because they are on the same network segment.

    The process for PE2 to forward packets from the CPE is the same as that for PE1 to forward packets from the CPE.

• Layer 3 packets transmitted across subnets

  • Uplink

    As shown in Figure 1-1070:

    Because the CPE is on a different network segment from PE1 and PE2, the destination MAC address of a Layer 3 unicast packet sent from CE1, CE2, or CE3 to the CPE is the MAC address of the BDIF interface on the Layer 3 gateway of PE1 or PE2. After receiving the packet, PE1 or PE2 removes the Layer 2 tag from the packet, searches for a matching Layer 3 routing entry, and obtains the outbound interface that is the BDIF interface connecting the CPE to the Layer 3 gateway. The BDIF interface searches the ARP table, obtains the destination MAC address, encapsulates the packet into a VXLAN packet, and sends it to the CPE through the VXLAN tunnel.

    After receiving the Layer 3 packet from PE1 or PE2, the CPE removes the Layer 2 tag from the packet because the destination MAC address is the MAC address of the BDIF interface on the CPE. Then the CPE searches the Layer 3 routing table to obtain a next-hop address to forward the packet.

  • Downlink

    As shown in Figure 1-1071:

    Before sending a Layer 3 unicast packet to CE1 across subnets, the CPE searches its Layer 3 routing table and obtains the outbound interface that is the BDIF interface on the Layer 3 gateway connecting to PE1. The BDIF interface searches the ARP table to obtain the destination MAC address, encapsulates the packet into a VXLAN packet, and forwards it to PE1 over the VXLAN tunnel.

    After receiving the packet from the CPE, PE1 removes the Layer 2 tag from the packet because the destination address of the packet is the MAC address of PE1's BDIF interface. Then PE1 searches the Layer 3 routing table and obtains the outbound interface that is the BDIF interface connecting PE1 to its attached CE. The BDIF interface searches its ARP table and obtains the destination address, performs Layer-2 encapsulation for the packet, and sends it to CE1.

    The process for PE2 to forward packets from the CPE is the same as that for PE1 to forward packets from the CPE.

Huawei's network functions virtualization infrastructure (NFVI) telco cloud solution incorporates Data Center Interconnect (DCI) and data center network (DCN) solutions. A large volume of UE traffic enters the DCN and accesses the vUGW and vMSE on the DCN. After being processed by the vUGW and vMSE, the UE traffic IPv4 or IPv6 is forwarded over the DCN to destination devices on the Internet. Likewise, return traffic sent from the destination devices to UEs also undergoes this process. To meet the preceding requirements and ensure that the UE traffic is load-balanced within the DCN, you need to deploy the NFVI distributed gateway function on DCN devices.

The vUGW is a unified packet gateway developed based on Huawei's CloudEdge solution. It can be used for 3rd Generation Partnership Project (3GPP) access in general packet radio service (GPRS), Universal Mobile Telecommunications System (UMTS), and Long Term Evolution (LTE) modes. The vUGW can function as a gateway GPRS support node (GGSN), serving gateway (S-GW), or packet data network gateway (P-GW) to meet carriers' various networking requirements in different phases and operational scenarios.

The vMSE is developed based on Huawei's multi-service engine (MSE). The carrier's network has multiple functional boxes deployed, such as the firewall box, video acceleration box, header enrichment box, and URL filtering box. All functions are added through patch installation. As time goes by, the network becomes increasingly slow, complicating service rollout and maintenance. To solve this problem, the vMSE integrates the functions of these boxes and manages these functions in a unified manner, providing value-added services for the data services initiated by users.

Networking Overview

Figure 1-1075 and Figure 1-1076 show NFVI distributed gateway networking. The DC gateways are the DCN's border gateways, which exchange Internet routes with the external network through PEs. L2GW/L3GW1 and L2GW/L3GW2 access the virtualized network functions (VNFs). VNF1 and VNF2 can be deployed as virtualized NEs to implement the vUGW and vMSE functions and connect to L2GW/L3GW1 and L2GW/L3GW2 through the interface processing unit (IPU).

This networking can be considered a combination of the distributed gateway function and VXLAN active-active/quad-active gateway function.

• The VXLAN active-active/quad-active gateway function is deployed on DC gateways. Specifically, a bypass VXLAN tunnel is established between DC gateways. All DC gateways use the same virtual anycast VTEP address to establish VXLAN tunnels with L2GW/L3GW1 and L2GW/L3GW2.

• The distributed gateway function is deployed on L2GW/L3GW1 and L2GW/L3GW2, and a VXLAN tunnel is established between them.

In the NFVI distributed gateway scenario, the NetEngine 8100 M, NetEngine 8000E M, NetEngine 8000 M can function as either a DC gateway or an L2GW/L3GW. However, if the NetEngine 8100 M, NetEngine 8000E M, NetEngine 8000 M is used as an L2GW/L3GW, east-west traffic cannot be balanced.

Each L2GW/L3GW in Figure 1-1075 represents two devices on the live network. anycast VXLAN active-active is configured on the devices for them to function as one to improve network reliability.

The method of deploying the VXLAN quad-active gateway function on DC gateways is similar to that of deploying the VXLAN active-active gateway function on DC gateways. This section uses the VXLAN active-active gateway function as an example.

Figure 1-1075 NFVI distributed gateway networking (active-active DC gateways)

Function Deployment

Generation of Forwarding Entries

In the NFVI distributed gateway networking, all traffic is forwarded at Layer 2 from DC gateways to VNFs after entering the DCN, regardless of whether it is from UEs to the Internet or vice versa. However, after traffic leaves the DCN, it is forwarded at Layer 3 from VNFs to DC gateways. This prevents traffic loops between DC gateways and L2GWs/L3GWs. On the network shown in Figure 1-1076, IPUs connect to multiple L2GWs/L3GWs. If Layer 3 forwarding is used between DC gateways and VNFs, some traffic forwarded by an L2GW/L3GW to the VNF will be forwarded to another L2GW/L3GW due to load balancing. For example, L2GW/L3GW2 forwards some of the traffic to L2GW/L3GW1 and vice versa. As a result, a traffic loop occurs. If Layer 2 forwarding is used, the L2GW/L3GW does not forward the Layer 2 traffic received from another L2GW/L3GW back, preventing traffic loops.

Figure 1-1076 Traffic loop

Traffic Forwarding Process

Figure 1-1083 shows the process of forwarding north-south traffic (from a UE to the Internet).

1. Upon receipt of UE traffic, the base station encapsulates these packets and redirect them to a GPRS tunneling protocol (GTP) tunnel whose destination address is the VNF IP address. The encapsulated packets reach the DC gateway through IP forwarding.

2. Upon receipt, the DC gateway searches its virtual routing and forwarding (VRF) table and finds a matching forwarding entry whose next hop is an IPU IP address and outbound interface is a VBDIF interface. Therefore, the received packets match the network segment on which the VBDIF interface resides. The DC gateway searches for the desired ARP entry on the network segment, finds a matching MAC forwarding entry based on the ARP entry, and recurses the route to a VXLAN tunnel based on the MAC forwarding entry. Then, the packets are forwarded to the L2GW/L3GW over a VXLAN tunnel.

3. Upon receipt, the L2GW/L3GW finds the target BD based on the L2 VNI, searches for a matching MAC forwarding entry in the BD, and then forwards the packets to the VNF based on the MAC forwarding entry.

4. After the packets reach the VNF, the VNF removes their GTP tunnel header, searches the routing table based on their destination IP addresses, and forwards them to the L2GW/L3GW through the VNF's default gateway.

5. After the packets reach the L2GW/L3GW, the L2GW/L3GW searches their VRF table for a matching forwarding entry. Over the default route advertised by the DC gateway to the L2GW/L3GW, the packets are encapsulated with the L3 VNI and then forwarded to the DC gateway through the VXLAN tunnel.

6. Upon receipt, the DC gateway searches the corresponding VRF table for a matching forwarding entry based on the L3 VNI and forwards these packets to the Internet.

Figure 1-1083 Process of forwarding north-south traffic from a UE to the Internet

Figure 1-1084 shows the process of forwarding north-south traffic from the Internet to a UE through the VNF.

1. A device on the Internet sends response traffic to a UE. The destination address of the response traffic is the destination address of the UE route. The route is advertised by the VNF to the DC gateway through the VPN BGP peer relationship, and the DC gateway in turn advertises the route to the Internet. Therefore, the response traffic must first be forwarded to the VNF first.

2. Upon receipt, the DC gateway searches the routing table for a forwarding entry that matches the UE route. The route is advertised over the VPN BGP peer relationship between the DC gateway and VNF and recurses to one or more VBDIF interfaces. Traffic is load-balanced among these VBDIF interfaces. A matching MAC forwarding entry is found based on the ARP information on these VBDIF interfaces. Based on the MAC forwarding entry, the response packets are encapsulated with the L2 VNI and then forwarded to the L2GW/L3GW over a VXLAN tunnel.

3. Upon receipt, the L2GW/L3GW finds the target BD based on the L2 VNI, searches for a matching MAC forwarding entry in the BD, obtains the outbound interface information from the MAC forwarding entry, and forwards these packets to the VNF.

4. Upon receipt, the VNF processes them and finds the base station corresponding to the destination address of the UE. The VNF then encapsulates tunnel information into these packets (with the base station as the destination) and forwards these packets to the L2GW/L3GW through the default gateway.

5. Upon receipt, the L2GW/L3GW searches its VRF table for the default route advertised by the DC gateway to the L2GW/L3GW. Then, the L2GW/L3GW encapsulates these packets with the L3 VNI and forwards them to the DC gateway over a VXLAN tunnel.

6. Upon receipt, the DC gateway searches its VRF table for the default (or specific) route based on the L3 VNI and forwards these packets to the destination base station. The base station then decapsulates these packets and sends them to the target UE.

Figure 1-1084 Process of forwarding north-south traffic from the Internet to a UE

During this process, the VNF may send the received packets to another VNF for value-added service processing, based on the packet information. In this case, east-west traffic is generated. Figure 1-1085 shows the process of forwarding east-west traffic (from VNF1 to VNF2), which differs from the north-south traffic forwarding process in packet processing after packets reach VNF1:

1. VNF1 sends a received packet to VNF2 for processing. VNF2 re-encapsulates the packet by using its own address as the destination address of the packet and sends the packet to the L2GW/L3GW over the default route.

2. Upon receipt, the L2GW/L3GW searches its VRF table and finds that multiple load-balancing forwarding entries exist. Some entries use the IPU as the outbound interface, and some entries use the L2GW/L3GW as the next hop.

3. If the path to the other L2GW/L3GW (L2GW/L3GW2) is selected preferentially, the packet is encapsulated with the L2 VNI and forwarded to L2GW/L3GW2 over a VXLAN tunnel. L2GW/L3GW2 finds the target BD based on the L2 VNI and the destination MAC address, and forwards the packet to VNF2.

4. Upon receipt, VNF2 processes the packet and forwards it to the Internet server. The subsequent forwarding process is the same as the process for forwarding north-south traffic.

Figure 1-1085 Process of forwarding east-west traffic (from VNF1 to VNF2)

Huawei's network functions virtualization infrastructure (NFVI) telco cloud solution incorporates Data Center Interconnect (DCI) and data center network (DCN) solutions. A large volume of UE traffic enters the DCN and accesses the vUGW and vMSE on the DCN. After being processed by the vUGW and vMSE, the UE traffic (IPv4 or IPv6) is forwarded over the DCN to destination devices on the Internet. Likewise, return traffic sent from the destination devices to UEs also undergoes this process. To meet the preceding requirements and ensure that the UE traffic is load-balanced within the DCN, you need to deploy the NFVI distributed gateway function on DCN devices.

The vUGW is a unified packet gateway developed based on Huawei's CloudEdge solution. It can be used for 3rd Generation Partnership Project (3GPP) access in general packet radio service (GPRS), Universal Mobile Telecommunications System (UMTS), and Long Term Evolution (LTE) modes. The vUGW can function as a gateway GPRS support node (GGSN), serving gateway (S-GW), or packet data network gateway (P-GW) to meet carriers' various networking requirements in different phases and operational scenarios.

The vMSE is developed based on Huawei's multi-service engine (MSE). The carrier's network has multiple functional boxes deployed, such as the firewall box, video acceleration box, header enrichment box, and URL filtering box. All functions are added through patch installation. As time goes by, the network becomes increasingly slow, complicating service rollout and maintenance. To solve this problem, the vMSE integrates the functions of these boxes and manages these functions in a unified manner, providing value-added services for the data services initiated by users.

Networking

Figure 1-1086 and Figure 1-1087 show NFVI distributed gateway networking. The DC gateways are the DCN's border gateways, which exchange Internet routes with the external network through PEs. L2GW/L3GW1 and L2GW/L3GW2 connect to virtualized network functions (VNFs). VNF1 and VNF2 can be deployed as virtualized NEs to respectively provide vUGW and vMSE functions and connect to L2GW/L3GW1 and L2GW/L3GW2 through interface processing units (IPUs).

This networking combines the distributed gateway function and the VXLAN active-active gateway function:

• The VXLAN active-active gateway function is deployed on DC gateways. Specifically, a bypass VXLAN tunnel is established between DC gateways. Both DC gateways use the same virtual anycast VTEP address to establish VXLAN tunnels with L2GW/L3GW1 and L2GW/L3GW2.

• The distributed gateway function is deployed on L2GW/L3GW1 and L2GW/L3GW2, and a VXLAN tunnel is established between L2GW/L3GW1 and L2GW/L3GW2.

In the NFVI distributed gateway scenario, the NetEngine 8100 M, NetEngine 8000E M, NetEngine 8000 M functions as either a DCGW or an L2GW/L3GW. However, if the NetEngine 8100 M, NetEngine 8000E M, NetEngine 8000 M is used as an L2GW/L3GW, east-west traffic cannot be balanced.

Each L2GW/L3GW in Figure 1-1086 represents two devices on the live network. anycast VXLAN active-active is configured on the devices for them to function as one to improve network reliability.

Figure 1-1086 NFVI distributed gateway networking (with active-active DC gateways)

Function Deployment

Generation of Forwarding Entries

Table 1-473 lists the differences between the asymmetric and symmetric modes in terms of forwarding entry generation.

Table 1-473 Differences between the asymmetric and symmetric modes in terms of forwarding entry generation

Asymmetric Mode

Symmetric Mode

All traffic is forwarded at Layer 2 from DC gateways to VNFs after entering the DCN, regardless of whether it is from UEs to the Internet or vice versa. However, after traffic leaves the DCN, it is forwarded at Layer 3 from VNFs to DC gateways. This prevents traffic loops between DC gateways and L2GWs/L3GWs. On the network shown in Figure 1-1087, IPUs connect to multiple L2GWs/L3GWs. If Layer 3 forwarding is used between DC gateways and VNFs, some traffic forwarded by an L2GW/L3GW to the VNF will be forwarded to another L2GW/L3GW due to load balancing. For example, L2GW/L3GW2 forwards some of the traffic to L2GW/L3GW1 and vice versa. As a result, a traffic loop occurs. If Layer 2 forwarding is used, the L2GW/L3GW does not forward the Layer 2 traffic received from another L2GW/L3GW back, preventing traffic loops.

After traffic enters the DCN, the traffic is forwarded from DC gateways to the VNF at Layer 3. The traffic from the VNF to DC gateways and then out of the DCN is also forwarded at Layer 3. On the network shown in Figure 1-1087, IPUs connect to multiple L2GWs/L3GWs. Layer 3 forwarding is used between DC gateways and VNFs, and some traffic forwarded by an L2GW/L3GW to the VNF will be forwarded over a VXLAN tunnel to another L2GW/L3GW due to load balancing. After receiving VXLAN traffic, an L2GW/L3GW searches for matching routes. If these routes work in hybrid load-balancing mode, the L2GW/L3GW preferentially selects the access-side outbound interface to forward the traffic, preventing loops.

Figure 1-1087 Traffic loop

Traffic Forwarding Process

Figure 1-1093 shows the process of forwarding north-south traffic (from a UE to the Internet).

1. Upon receipt of UE traffic, the base station encapsulates these packets and redirect them to a GPRS tunneling protocol (GTP) tunnel whose destination address is the VNF IP address. The encapsulated packets reach the DC gateway through IP forwarding.

2. After receiving these packets, the DC gateway searches the VRF table and finds that the next hop of the forwarding entry corresponding to the VNF address is an IPU address and the outbound interface is a VXLAN tunnel. The DC gateway then performs VXLAN encapsulation and forwards the packets to the L2GW/L3GW at Layer 3.

3. Upon receipt of these packets, the L2GW/L3GW finds the corresponding VPN instance based on the L3 VNI, searches for a matching route in the VPN instance's routing table based on the VNF address, and forwards the packets to the VNF.

4. After the packets reach the VNF, the VNF removes their GTP tunnel header, searches the routing table based on their destination IP addresses, and forwards them to the L2GW/L3GW through the VNF's default gateway.

5. After the packets reach the L2GW/L3GW, the L2GW/L3GW searches their VRF table for a matching forwarding entry. Over the default route advertised by the DC gateway to the L2GW/L3GW, the packets are encapsulated with the L3 VNI and then forwarded to the DC gateway through the VXLAN tunnel.

6. Upon receipt, the DC gateway searches the corresponding VRF table for a matching forwarding entry based on the L3 VNI and forwards these packets to the Internet.

Figure 1-1093 Process of forwarding north-south traffic from a UE to the Internet

Figure 1-1094 shows the process of forwarding north-south traffic from the Internet to a UE through the VNF.

1. A device on the Internet sends response traffic to a UE. The destination address of the response traffic is the destination address of the UE route. The route is advertised by the VNF to the DC gateway through the VPN BGP peer relationship, and the DC gateway in turn advertises the route to the Internet. Therefore, the response traffic must first be forwarded to the VNF first.

2. After the response traffic reaches the DC gateway, the DC gateway searches the routing table for forwarding entries corresponding to UE routes. These routes are learned by the DC gateway from the VNF over the VPN BGP peer relationship. These routes finally recurse to VXLAN tunnels, the response packets are encapsulated into VXLAN packets and forwarded to the L2GW/L3GW at Layer 3.

3. After these packets reach the L2GW/L3GW, the L2GW/L3GW finds the corresponding VPN instance based on the L3 VNI, searches for a route corresponding to the UE address in the VPN instance's routing table, and forwards these packets to the VNF.

4. Upon receipt, the VNF processes them and finds the base station corresponding to the destination address of the UE. The VNF then encapsulates tunnel information into these packets (with the base station as the destination) and forwards these packets to the L2GW/L3GW through the default gateway.

5. Upon receipt, the L2GW/L3GW searches its VRF table for the default route advertised by the DC gateway to the L2GW/L3GW. Then, the L2GW/L3GW encapsulates these packets with the L3 VNI and forwards them to the DC gateway over a VXLAN tunnel.

6. Upon receipt, the DC gateway searches its VRF table for the default (or specific) route based on the L3 VNI and forwards these packets to the destination base station. The base station then decapsulates these packets and sends them to the target UE.

Figure 1-1094 Process of forwarding north-south traffic from the Internet to a UE

During this process, the VNF may send the received packets to another VNF for value-added service processing, based on the packet information. In this case, east-west traffic is generated. Figure 1-1095 shows the process of forwarding east-west traffic (from VNF1 to VNF2), which differs from the north-south traffic forwarding process in packet processing after packets reach VNF1:

1. VNF1 sends a received packet to VNF2 for processing. VNF2 re-encapsulates the packet by using its own address as the destination address of the packet and sends the packet to the L2GW/L3GW1 over the default route.

2. Upon receipt, the L2GW/L3GW1 searches its VRF table and finds that multiple load-balancing routes exist. Some routes use the IPU as the outbound interface, and some routes use L2GW/L3GW2 as the next hop.

3. If these routes work in hybrid load-balancing mode, L2GW/L3GW1 preferentially selects only the routes with the outbound interfaces being IPUs and steers packets to VNF2 to prevent loops. If these routes do not work in hybrid load-balancing mode, L2GW/L3GW1 forwards packets in load-balancing route. Packets are encapsulated into VXLAN packets before they are sent to L2GW/L3GW2 at Layer 2. After these packets reach L2GW/L3GW2, L2GW/L3GW2 finds the corresponding BD based on the L2 VNI, then finds the destination MAC address, and finally forwards these packets to VNF2.

4. Upon receipt, VNF2 processes the packet and forwards it to the Internet server. The subsequent forwarding process is the same as the process for forwarding north-south traffic.

Figure 1-1095 Process of forwarding east-west traffic (from VNF1 to VNF2)

Service Description

Currently, data centers are expanding on a large scale for enterprises and carriers, with increasing deployment of virtualization and cloud computing. In addition, to accommodate more services while reducing maintenance costs, data centers are employing large Layer 2 and virtualization technologies.

As server virtualization is implemented in the physical network infrastructure for data centers, VXLAN, an NVO3 technology, has adapted to the trend by providing virtualization solutions for data centers.

Networking Description

On the network shown in Figure 1-1096, an enterprise has VMs deployed in different data centers. Different network segments run different services. The VMs running the same service or different services in different data centers need to communicate with each other. For example, VMs of the financial department residing on the same network segment need to communicate, and VMs of the financial and engineering departments residing on different network segments also need to communicate.

Figure 1-1096 Communication between terminal users on a VXLAN

Feature Deployment

Configure VXLAN on devices to trigger VXLAN tunnel establishment and dynamic learning of ARP and MAC address entries. By now, terminal users on the same network segment and different network segments can communicate through the Layer 2 and Layer 3 VXLAN gateways based on ARP and routing entries.

Service Description

Currently, data centers are expanding on a large scale for enterprises and carriers, with increasing deployment of virtualization and cloud computing. In addition, to accommodate more services while reducing maintenance costs, data centers are employing large Layer 2 and virtualization technologies.

As server virtualization is implemented in the physical network infrastructure for data centers, VXLAN, an NVO3 technology, has adapted to the trend by providing virtualization solutions for data centers, allowing intra-VXLAN communication and communication between VXLANs and legacy networks.

Networking Description

On the network shown in Figure 1-1097, an enterprise has VMs deployed for the finance and engineering departments and a legacy network for the human resource department. The finance and engineering departments need to communicate with the human resource department.

Figure 1-1097 Communication between terminal users on a VXLAN and legacy network

Feature Deployment

As shown in Figure 1-1097:

Deploy Device 2 as Layer 2 VXLAN gateway and Device 3 as a Layer 3 VXLAN gateway. The VXLAN gateways are VXLANs' edge devices connecting to legacy networks and are responsible for VXLAN encapsulation and decapsulation. Establish a VXLAN tunnel between Device 2 and Device 3 for VXLAN packet transmission.

Service Description

Enterprises configure server virtualization on DCNs to consolidate IT resources, improve resource use efficiency, and reduce network costs. With the wide deployment of server virtualization, an increasing number of VMs are running on physical servers, and many applications are running in virtual environments, which bring great challenges to virtual networks.

Network Description

On the network shown in Figure 1-1098, an enterprise has two servers in the DC: engineering and finance departments on Server1 and the marketing department on Server2.

The computing space on Server1 is insufficient, but Server2 is not fully used. The network administrator wants to migrate the engineering department to Server2 without affecting services.

This scenario applies to IPv4 over IPv4, IPv6 over IPv4, IPv4 over IPv6, and IPv6 over IPv6 networks. Figure 1-1098 shows an IPv4 over IPv4 network.

Figure 1-1098 Department distribution

Feature Deployment

To ensure uninterrupted services during the migration of the engineering department, the IP and MAC addresses of the engineering department must remain unchanged. This requires that the two servers belong to the same Layer 2 network. If conventional migration methods are used, the administrator may have to purchase additional physical devices to distribute traffic and reconfigure VLANs. These methods may also result in network loops and additional system and management costs.

VXLAN can be used to migrate the engineering department to Server2. VXLAN is a network virtualization technology that uses MAC-in-UDP encapsulation. This technology can establish a large Layer 2 network connecting all terminals with reachable IP routes, as long as the physical network supports IP forwarding.

The engineering department is migrated to Server2 through the VXLAN tunnel. Online users are unaware of the migration. After the engineering department is migrated from Server1 to Server2, terminals send gratuitous ARP or RARP packets to update all gateways' MAC addresses and ARP entries of the original VMs to those of the VMs to which the R&D department is migrated.