SR-IOV in VMware Applications

The following describes how to apply SR-IOV on the MZ510 in VMware. You can create multiple VFs on the MZ510 and allocate them to VMs.

You have installed ESXi 5.1 on the compute node and the vSphere Client on the PC.

Access the Emulex BIOS, and enable SR-IOV on the MZ510 port.

Before using the SR-IOV function on the MZ510, disable UMC.

Run the SSH terminal tool to log in to the ESXi CLI, load the be2net driver, and set the VF parameter (the number of VFs) by using the following command:

~# esxcli system module parameters set -m be2net -p "max_vfs=x,y"

Or:

~# esxcfg-module be2net –s max_vfs=x,y

The parameter x indicates the number of VFs of Port0 (corresponding to vmnic0), and y indicates the number of VFs of Port1 (corresponding to vmnic1). In VMware, each physical port (PF) supports a maximum of 16 VFs.

After the configuration is complete, restart the ESXi for the VFs to take effect. When a vmnic functioning as a PF virtualizes VFs, it is not displayed on the Network Adapter screen. For example, vmnic0 is not displayed, as shown in Figure 5-36. On the vSphere Client, click the Configuration tab. On the Configuration tab page, open the Network Adapter screen. vmnic0 is also not displayed, as shown in Figure 5-37. To ensure that the management network is available, the port (for example, vmnic1) connected to the management network port does not generate a VF or its SR-IOV is not enabled. Therefore, run the following command to generate the VF of vmnic0:

~# esxcli system module parameters set -m be2net -p "max_vfs=16,0"

Or:

~# esxcfg-module be2net –s max_vfs=16,0

Figure 5-36 vmnic0 that is not displayed on the ESXi Network Adapter screen after it generates a VF

Figure 5-37 vmnic0 that is not displayed on the vSphere Client after it generates a VF

If a compute node is equipped with multiple MZ510s, run the following command to configure VF parameters on each MZ510 port:

~# esxcli system module parameters set -m be2net -p "max_vfs=x1,y1,x2,y2,x3,y3,…"

Or:

~# esxcfg-module be2net –s max_vfs=x1,y1,x2,y2,x3,y3,…

The parameter x1 indicates the number of VFs of Port0 on the first MZ510, y1 indicates the number of VFs of Port1 on the first MZ510, x2 indicates the number of VFs of Port0 on the second MZ510, y2 indicates the number of VFs of Port1 on the second MZ510, x3 indicates the number of VFs of Port0 on the third MZ510, and y3 indicates the number of VFs of Port1 on the third MZ510.

On the ESXi CLI, you can run commands to view the VF configuration and generation results, as shown in Figure 5-38.

Check the number of VFs.

~# esxcfg-module –g be2net

Check the NICs (PFs) where SR-IOV is enabled.

~# esxcli network sriovnic list

Check VF status in the PF.

~# esxcli network sriovnic vf list –n vmnic[num]

Figure 5-38 Checking the VF generation result

On the vSphere Client, click the Configuration tab. On the Configuration tab page, click Advanced Settings. You can see that vmnic0 is virtualized into 16 PCIe pass-through devices, as shown in Figure 5-39.

Figure 5-39 vmnic0 displayed as DirectPath I/O on the vSphere Client after it generates VFs

On the ESXi CLI, you can see the PCI devices corresponding to the VFs, as shown in Figure 5-40.

Figure 5-40 PCI devices corresponding to the VFs

Create a VM, and assign VFs to the VM. In VMware, each VM supports a maximum of six VFs.

Select a VM, click Edit Settings, click Add, and select PCI Device, as shown in Figure 5-41. Select a VF and assign it to the specified VM, as shown in Figure 5-42.

Figure 5-41 Adding a VF to the VM (1)

Figure 5-42 Adding a VF to the VM (2)

After a VF is added, you can view the VF (PCI device) in the Edit Settings window of the VM, as shown in Figure 5-43.

Figure 5-43 VF associated with the VM

The following uses SLES 11 SP2 on the VM as an example. You can see four NICs in the OS. eth0 and eth1 correspond to vmnic1, and belong to the virtual port group VM Network. eth2 and eth3 correspond to two VFs, as shown in Figure 5-44.

Figure 5-44 NIC ports of each VF in the VM OS

Set IP addresses for the NIC ports: 192.168.2.181 for eth0, 192.168.2.182 for eth1, 192.168.1.211 for eth2, and 192.168.2.211 for eth3. Then the VM can communicate with the management network and the external network, as shown in Figure 5-45.

Figure 5-45 Communication between the VM and the external network through the VF

The ports eth0 and eth1 connect to the management network. vmnic1 is used as an example. The ports eth2 and eth3 correspond to vmnic0 (virtualized VF).

You are advised to make ports eth1 and eth2 correspond to vmnic0 and vmnic1 respectively and assign different IP address segments for the ports. Create another VM (VM_2), allocate a VF to it, install SLES 11 SP2 on the VM, and set an IP address for the NIC port corresponding to the VF. Then, VM_2 can communicate with VM_1, as shown in Figure 5-46.

Figure 5-46 Communication between VMs through their VFs