What Is EVPN? How Does BGP EVPN Work?

Table 1-1 lists the five types of EVPN routes defined in EVPN NLRI. Type 1 to Type 4 routes are defined in RFC 7432, and Type 5 route is defined in a later draft.

Type 1 and Type 4 routes are used in EVPN Ethernet Segment Identifier (ESI) all-active scenarios. The EVPN ESI all-active function allows multi-homing and all-active VXLAN gateways to be deployed based on RFC standards, effectively improving the reliability on the VXLAN access side. Currently, only some CloudEngine switch models support this function. For details, see "EVPN ESI All-Active Function" in the product documentation of CloudEngine switches.

This document focuses on the common Type 2, Type 3, and Type 5 EVPN routes.

A VXLAN tunnel is determined by a pair of VTEPs. In an intra-subnet communication scenario, a VXLAN tunnel can be established between two VTEPs when they have reachable routes to each other's IP address because they only need to communicate in the same Layer 2 BD. When EVPN is used to dynamically establish a VXLAN tunnel, two VTEPs establish a BGP EVPN peer relationship and exchange Type 3 routes to transmit VNI and VTEP IP address information. A VXLAN tunnel is then dynamically established between them.

The following uses Figure 1-5 as an example to describe how VXLAN tunnels are established between VTEPs through Type 3 routes.

Figure 1-5 Intra-subnet VXLAN tunnel establishment

In the figure, Leaf1, Leaf2, and Leaf3 function as VTEPs, and Leaf1 advertises routes to Leaf2 and Leaf3.

1. After the VTEP IP address, L2VNI, and EVPN instance are configured on Leaf1 (as shown in the following example command output), Leaf1 advertises EVPN Type 3 routes to Leaf2 and Leaf3. The routes carry information including the L2VNI, local VTEP IP address, RD of the EVPN instance, and export route target (ERT).

   [Leaf1]
   bridge-domain 10
    vxlan vni 10    //L2VNI
    evpn
     route-distinguisher 1:10    //RD of the EVPN instance
     vpn-target 0:10 export-extcommunity    //ERT of the EVPN instance
     vpn-target 100:5000 export-extcommunity
     vpn-target 0:10 import-extcommunity
   #
   interface Nve1
    source 1.1.1.1   //VTEP IP address of Leaf1
    vni 10 head-end peer-list protocol bgp
   #

2. After Leaf2 and Leaf3 receive Type 3 routes from Leaf1, they establish Layer 2 VXLAN tunnels to Leaf1 if there are reachable routes to the VTEP IP address of Leaf1. In addition, if the VNI in the routes is the same as the local VNI, an ingress replication list is created for forwarding broadcast, multicast, and unknown unicast packets.

   After Leaf2 and Leaf3 receive EVPN routes from Leaf1, they determine whether to accept the routes based on whether the RT (ERT of the EVPN instance) carried in the routes matches the import route target (IRT) of the local EVPN instance.

After the preceding process, Leaf2 and Leaf3 can create an ingress replication list to Leaf1 to guide the forwarding of BUM packets. Similarly, Leaf1 also creates an ingress replication list to Leaf2 and Leaf3.

When EVPN is used as the control plane of VXLAN, MAC address learning through EVPN can replace MAC address learning through flooding on the data plane, reducing flooding traffic. VTEPs transmit Type 2 routes to learn MAC addresses through EVPN.

The following uses Figure 1-6 as an example to describe how VTEPs learn MAC addresses of remote hosts through EVPN.

Figure 1-6 Learning MAC addresses of remote hosts through EVPN

In the figure, Leaf1 and Leaf2 function as VTEPs and connect to Host1 and Host2 on the same network segment, respectively. Leaf1 advertises a Type 2 route to Leaf2.

1. When Host1 is connected to Leaf1, it sends ARP and DHCP packets to Leaf1. Leaf1 learns the MAC address of Host1 through the packets and records it in the local MAC address table.

   After learning the MAC address entry of the local host, Leaf1 advertises an EVPN Type 2 route to its peer Leaf2. The route carries the ERT of the local EVPN instance, VTEP IP address, L2VNI, and MAC address of Host1. The ERT of the local EVPN instance, VTEP IP address, and L2VNI are obtained from the local VTEP configuration. The following is an example:

   [Leaf1]
   bridge-domain 10
    vxlan vni 10    //L2VNI
    evpn
     route-distinguisher 10:1
     vpn-target 0:10 export-extcommunity    //ERT of the EVPN instance
     vpn-target 100:5000 export-extcommunity
     vpn-target 0:10 import-extcommunity
   #
   interface Nve1
    source 1.1.1.1   //VTEP IP address of Leaf1
    vni 10 head-end peer-list protocol bgp
   #

2. After receiving the Type 2 route from Leaf1, Leaf2 learns the MAC address of Host1 and saves it in the MAC address table. The next hop is the VTEP IP address of Leaf1.

   Note that Leaf2 determines whether to accept the EVPN route received from Leaf1 based on the route target (RT) of the EVPN instance. An RT is a BGP extended community attribute used to control the advertisement and acceptance of EVPN routes. That is, RTs determine whether the local and peer ends can accept EVPN routes from each other.

   There are two types of RTs:

   • Export RT (ERT): When the local end advertises an EVPN route, the ERT is carried in the route.
   • Import RT (IRT): When receiving an EVPN route from a peer, the local end compares the ERT carried in the route with the local IRT. The local end accepts the route only when the ERT and IRT are the same. Otherwise, the local end discards the route.

   In this example, Leaf2 accepts the EVPN route from Leaf1 because the IRT configured on Leaf2 is the same as the ERT configured on Leaf1. That is, the IRT of the EVPN instance configured on Leaf2 is 0:10, which is the same as the ERT configured on Leaf1.

   [Leaf2]
   bridge-domain 10
    vxlan vni 10    //L2VNI
    evpn
     route-distinguisher 10:2
     vpn-target 0:10 export-extcommunity    
     vpn-target 100:5000 export-extcommunity
     vpn-target 0:10 import-extcommunity   //IRT of the EVPN instance
   #

After the preceding process, Leaf2 can learn the MAC address of Host1 without sending a broadcast request packet. Similarly, Leaf1 can learn the MAC address of Host2.

Note that EVPN only reduces traffic flooding on the network, but cannot completely prevent it, for example, in the following scenarios:

1. If a silent host exists on the network, the host does not send ARP and DHCP packets. As a result, the VTEP connected to the host cannot learn the MAC address of the host and cannot send MAC address information of the host to other VTEPs.
2. When a host communicates with another device for the first time, the host broadcasts an ARP Request packet, causing traffic flooding. In this case, ARP broadcast suppression can be used to prevent flooding. For details, see ARP Broadcast Suppression on a VXLAN BGP EVPN Network in this document.

Figure 1-7 Host route advertisement using EVPN

In the figure, Leaf1 and Leaf2 function as VTEPs and Layer 3 gateways, and connect to Host1 and Host2 on different network segments, respectively. Leaf1 advertises a route to Leaf2.

1. When Host1 is connected to Leaf1, it sends ARP and DHCP packets to Leaf1. Leaf1 learns the ARP entry of Host1 through the packets. Leaf1 can also obtain the L2VNI, L3VPN instance, and L3VNI associated with the L3VPN instance based on the BD to which Host1 belongs.

   Why are there the L3VPN and L3VNI? Servers of multiple tenants may connect to the same leaf node. To isolate tenants, different L3VPNs are created on the leaf node to isolate the routing tables of different tenants. In this way, routes of different tenants are stored in different VRF routing tables. L3VNIs are used to identify these L3VPNs. When the leaf node receives a data packet carrying an L3VNI from a peer, it finds the corresponding L3VPN based on the L3VNI, and searches the routing table of the L3VPN instance to forward the packet.

   The following example command output shows key configurations used by Leaf1 to obtain the L2VNI, L3VPN instance, and L3VNI associated with the L3VPN instance.

   [Leaf1]
   ip vpn-instance vpn1    //L3VPN instance
    ipv4-family
     route-distinguisher 20:4
     vpn-target 100:5000 export-extcommunity evpn
     vpn-target 100:5000 import-extcommunity evpn
    vxlan vni 5000      //L3VNI associated with the L3VPN instance
   #
   bridge-domain 10
    vxlan vni 10     //L2VNI
    evpn
     route-distinguisher 10:4
     vpn-target 0:10 export-extcommunity
     vpn-target 100:5000 export-extcommunity
     vpn-target 0:10 import-extcommunity
   #
   interface Vbdif10    //Obtain the Layer 3 VBDIF interface and the L3VPN instance bound to the interface based on BD information.
    ip binding vpn-instance vpn1 
    ip address 192.168.1.1 255.255.255.0
    mac-address 0000-5e00-0102
    vxlan anycast-gateway enable
    arp collect host enable
   #

   In conclusion, Leaf1 obtains the following information about Host1: IP address, MAC address, L2VNI, and L3VNI associated with the L3VPN instance bound to the VBDIF interface.

   1. Leaf1 can generate an EVPN Type2 route (see the table in the preceding figure) for the EVPN instance based on the information. In addition to the obtained information about Host1, the route also carries information including the ERT of the local EVPN instance, the next hop (the local VTEP's IP address) of the route, and the VTEP's MAC address. Leaf1 advertises the route to its peer Leaf2.
   2. The EVPN instance on Leaf1 sends Host1's IP address, MAC address, and L3VNI to the local L3VPN instance so that a route to Host1 is generated in the local L3VPN instance.
2. After receiving the Type 2 route from Leaf1, Leaf2 learns the IP address of Host1, saves it in the corresponding routing table, and records the corresponding L3VNI. The next hop of the route is the VTEP IP address of Leaf1. The details are as follows:
   1. Check whether the ERT in the route is the same as the IRT of the EVPN instance on the receiver. If they are the same, Leaf2 accepts the route and obtains the host IP address and MAC address from the route based on the EVPN instance for host ARP advertisement.
   2. Check whether the ERT in the route is the same as the IRT of the L3VPN instance on the receiver (as shown in the example in the following table). If they are the same, Leaf2 accepts the route, obtains the host IP address and L3VNI from the route based on the L3VPN instance, and generates a route to Host1 in the routing table of the L3VPN instance. The next hop of the route is set to the VXLAN tunnel interface of Leaf1.

      Leaf1 (Sender)	Leaf2 (Receiver)
      ip vpn-instance vpn1 ipv4-family route-distinguisher 20:2 vpn-target 100:5000 export-extcommunity evpn vpn-target 100:5000 import-extcommunity evpn vxlan vni 5000 # bridge-domain 10 vxlan vni 10 evpn route-distinguisher 10:2 vpn-target 100:10 export-extcommunity vpn-target 100:5000 export-extcommunity //ERT of the EVPN instance on the sender vpn-target 100:10 import-extcommunity #	ip vpn-instance vpn1 ipv4-family route-distinguisher 20:3 vpn-target 100:5000 export-extcommunity evpn vpn-target 100:5000 import-extcommunity evpn //IRT (eIRT) of the L3VPN instance on the receiver vxlan vni 5000 # bridge-domain 20 vxlan vni 20 evpn route-distinguisher 10:3 vpn-target 100:20 export-extcommunity vpn-target 100:5000 export-extcommunity vpn-target 100:20 import-extcommunity #

   3. After receiving the route, Leaf2 obtains the VTEP IP address of Leaf1 based on the next hop in the accepted route for the EVPN instance or L3VPN instance. If there is a reachable route to the IP address, a VXLAN tunnel to Leaf1 is established.

After the preceding process, Leaf2 can learn the IP route to Host1 and forward packets to Host1 based on the routing table. Similarly, Leaf1 can learn the IP route to Host2.

Figure 1-8 Network segment route advertisement using EVPN

The following uses Figure 1-8 as an example to describe how VTEPs advertise network segment routes. In the figure, Leaf1 and Leaf2 function as VTEPs and Layer 3 gateways. Leaf1 is connected to the network segment 192.168.1.0/24.

1. Leaf1 saves the local IP network segment route and advertises the route to Leaf2 through an EVPN Type 5 route. The route carries information including the IP prefix, mask length, and L3VNI of the corresponding VRF (as shown in the table in the preceding figure).
2. After receiving the Type 5 route from Leaf1, Leaf2 learns the IP network segment route, saves it in the corresponding routing table, and records the corresponding L3VNI. The next hop of the route is the VTEP IP address of Leaf1.

   After receiving the EVPN route from Leaf1, Leaf2 checks whether the RT carried in the EVPN route matches the IRT of the local L3VPN instance to determine whether to add the network segment route to the corresponding VRF routing table. (The RT in the Type 5 route is the ERT of the L3VPN instance.) If the IRT of a VRF is the same as the RT carried in the EVPN route, Leaf2 accepts the route, obtains network segment route and L3VNI information, and generates a network segment route in the routing table. The next hop of the route is set to the VTEP IP address of Leaf1. In addition, if there is a reachable route to the VTEP IP address of Leaf1, a VXLAN tunnel to Leaf1 is established.

   Leaf1 (Sender)

   Leaf2 (Receiver)

   ip vpn-instance vpn1 ipv4-family route-distinguisher 20:2 vpn-target 100:5000 export-extcommunity evpn //The ERT of the sender in a Type 5 route is the ERT (eERT) of the L3VPN instance. vpn-target 100:5000 import-extcommunity evpn vxlan vni 5000 # bridge-domain 10 vxlan vni 10 evpn route-distinguisher 10:2 vpn-target 100:10 export-extcommunity vpn-target 100:5000 export-extcommunity vpn-target 100:10 import-extcommunity #

   ip vpn-instance vpn1 ipv4-family route-distinguisher 20:3 vpn-target 100:5000 export-extcommunity evpn vpn-target 100:5000 import-extcommunity evpn //IRT (eIRT) of the L3VPN instance on the receiver vxlan vni 5000 # bridge-domain 20 vxlan vni 20 evpn route-distinguisher 10:3 vpn-target 100:20 export-extcommunity vpn-target 100:5000 export-extcommunity vpn-target 100:20 import-extcommunity #

After the preceding process, Leaf2 can learn the network segment route of Leaf1 and forward packets to the network segment based on the routing table.

Intra-subnet packet forwarding is completed only between Layer 2 VXLAN gateways. Layer 3 VXLAN gateways do not need to be aware of the process.

Intra-Subnet Known Unicast Packet Forwarding

As shown in Figure 1-9, Host1 and Host2 are on the same subnet. The following uses the scenario where Host1 sends known unicast packets to Host2 as an example to describe the packet forwarding process on the VXLAN network.

Figure 1-9 Intra-subnet known unicast packet forwarding

1. Host1 sends a packet destined for Host2. If Host1 does not have the MAC address of Host2, Host1 sends broadcast ARP Request packets to obtain the MAC address of Host2. This process is not described in detail here. It is assumed that Host1 has obtained the MAC address of Host2.
2. After receiving a packet from Host1, Leaf1 determines the BD to which Host1 belongs based on the inbound interface or VLAN information of the packet and searches for the outbound interface in the BD. (As described in MAC Address Learning Through EVPN, Leaf1 learns the MAC address of Host2 and the outbound interface is VTEP 2.2.2.2.) Leaf1 then encapsulates the packet into a VXLAN packet and forwards it.
3. After receiving the VXLAN packet, Leaf2 obtains the Layer 2 BD based on the VNI in the packet and performs VXLAN decapsulation to obtain the inner Layer 2 packet.

4. Leaf2 searches the local MAC address table for the outbound interface based on the destination MAC address of the inner packet, and then forwards the packet to Host2.

The process of sending packets from Host2 to Host1 is the same as the preceding process.

Intra-Subnet BUM Packet Forwarding

When receiving a BUM packet from a host to another host on the same subnet, a VTEP sends the packet to all VTEPs connecting to hosts on the same subnet.

In the example shown in Figure 1-10, Host1 sends a broadcast packet. After receiving the broadcast packet from Host1, Leaf1 determines the BD to which Host1 belongs based on the inbound interface or VLAN information of the packet, searches for the list of all tunnels in the BD, encapsulates the packet based on the obtained tunnel list, and sends the packet through all tunnels. In this way, the packet is forwarded to Host2 and Host3 on the same subnet.

Figure 1-10 Intra-subnet BUM packet forwarding

In a distributed gateway scenario shown in Figure 1-11, Leaf1 and Leaf2 function as Layer 3 VXLAN gateways and perform VXLAN encapsulation and Layer 3 forwarding. The spine node functions only as a VXLAN packet forwarding node and does not process VXLAN packets.

Figure 1-11 Inter-subnet packet forwarding in a distributed gateway scenario

The following uses the scenario where Host1 sends packets to Host2 as an example to describe the packet forwarding process on the VXLAN network.

1. Since Host1 and Host2 belong to different network segments, Host1 first sends a packet to the gateway (Leaf1) to forward it.
2. After receiving a packet from Host1, Leaf1 determines that Layer 3 forwarding is required based on the destination address of the packet. Leaf1 determines the BD to which Host1 belongs based on the inbound interface or VLAN information of the packet, finds the L3VPN instance bound to the BD, and searches the routing table of the L3VPN instance. As described in Inter-Subnet VXLAN Tunnel Establishment and Route Advertisement, Leaf1 can learn the host route of Host2 in the distributed gateway scenario.

   Leaf1 obtains the L3VNI and next hop based on the route, performs VXLAN encapsulation, and forwards the packet to Leaf2.

3. After receiving the VXLAN packet, Leaf2 decapsulates the packet and finds the corresponding L3VPN instance based on the Layer 3 VNI carried in the packet. Then, Leaf2 searches the routing table of the L3VPN instance and finds that the next hop of the packet is the gateway interface address. Leaf2 replaces the destination MAC address with the MAC address of Host2, replaces the source MAC address with the MAC address of Leaf2, and forwards the packet to Host2.

The process of sending packets from Host2 to Host1 is the same as the preceding process.