No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

RH5885 V3-High I/O Latency of Servers

Publication Date:  2016-06-23 Views:  114 Downloads:  0
Issue Description
The customer reported that alarms indicating high I/O latency were generated for VMware 5.5 on RH5885 V3 servers.
Alarm Information

Handling Process

1 Fault Analysis

1.1 Alarm Information

In March 2016, the customer reported that alarms indicating high I/O latency were generated for VMware 5.5 on RH5885 V3 servers.

Figure 1-1 Alarms for the TSPIIV1 server

 

Figure 1-2 Alarms for the TSPIIV3 server

 

1.2 Fault Analysis

1.2.1 VMware Found That the Log File Contained a Large Number of Logs Indicating That the FCP Command of the lpfc Driver That Comes with VMware Failed, and Inferred That the lpfc Driver Caused High I/O Latency.

The vmkwarning.log file of VMware contained a large number of logs indicating that the FCP command of the lpfc driver that comes with VMware failed.

Figure 1-3 Logs indicating an FCP command failure

 

According to the log analysis result, VMware inferred that the lpfc driver that comes with VMware caused the FCP command to fail, which then triggered alarms indicating high I/O latency.

Figure 1-4 Official response from VMware

 

The customer uses the LPFC driver 10.0.100.1-1vmw, which comes with VMware.

Compatible driver version at the VMware website:

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=5080&releaseid=271&deviceCategory=io&details=1&partner=31&deviceTypes=5&page=3&display_interval=10&sortColumn=Partner&sortOrder=Asc

Figure 1-5 VMware driver compatibility information

 

1.2.2 Server Hardware Was Normal, Which Did Not Cause High I/O Latency

According to the hardware logs of the three servers returned from the site, the server hardware was normal and no alarm was generated.

Figure 1-6 Server hardware logs

Root Cause

1 Conclusion

The lpfc driver that comes with VMware caused high I/O latency.

Solution

Solution:

1.         Upgrade the LPFC driver to 11.0.237.0.

Driver download link:

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-EMULEX-LPFC-1102370&productId=353#

 

2.         Upgrade the firmware for the 8G FC card lpe12002.

http://docs.avagotech.com/docs/13357157

END