No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Network Interruption Due to the Incompatibility Between RedHat&CentOS and Broadcom5709

Publication Date:  2015-06-25 Views:  87 Downloads:  0
Issue Description
For some RH2285s, services are interrupted because eth0 network port is irregularly disconnected from the network. Services are restored when the operating system (OS) restarts.

Handling Process
To disable the P and C state for the basic input/output system (BIOS) and the MSI function for bnx2, perform the following operations:

1.  To disable the P and C state for the BIOS, perform the following operations:
  • Press Delete in the power-on self test (POST) phase to open the BIOS.
  • Choose Advanced > CPU Configuration. In the CPU Configuration interface, set Intel® SpeedStep(tm) tech (that is, the P state) and Intel ® C-STATE tech (that is, the C state) to Disabled. Then press F10 to save the settings and exit. For details, see Figure 2 and Figure 3.
Figure 2 Enabling the P or C state



Figure 3 Disabling the P or C state



2.  Perform the following operations to disable the MSI function of bnx2 for Broadcom 5709 (the OS is the 64-bit CentOS5.4, and the kernel version is 2.6.18-164.el5):
  • Log in to the server OS, run the terminal, and enter vi /etc/modprobe.conf. The text compiling interface is displayed.
  • Enter options bnx2 disable_msi=1 at the end of the text (shown in Figure 4), save the setting and exit, and restart the system.
Figure 4 Modifying the driver configuration file



3.  After restarting the system, enter the grep –i eth /proc/interrupts command on the terminal. Check whether MSI is disabled based on the command output (if MSI is disabled, PCI-MSI-X is not displayed), as shown in Figure 5 and Figure 6.

Figure 5 Enabled MSI

Root Cause
1.  CentOS used by the RH2285 is incompatible with network interface cards (NICs).

Based on the onsite feedback, the onsite OS is the 64-bit CentOS5.4, and the kernel version is 2.6.18-164.el5.

Linux version 2.6.18-164.el5 (mockbuild@builder10.centos.org)
(gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 03:28:30 EDT 2009


CentOS is the free and compiled version of the RedHat OS, and is equivalent to the RedHat OS. After communicating with the RedHat OS manufacturer and the NIC manufacturer, Huawei engineers learn that the RedHat OS in 5.3 or later versions has a compatibility bug for supporting the bnx2 driver of Broadcom 5709. In certain special scenarios, when the service data flow is too large (that is, the network port load is too high), NICs fail with a small possibility, which causes service interruption. For details about the problem description (shown in Figure 1), visit http://kbase.redhat.com/faq/docs/DOC-26837.

Figure 1 Incompatibility between RHEL5.3 and BCM5709




Based on the description in the preceding figure, the incompatibility problem persists in servers that use Broadcom 5709 chips in the RedHat and CentOS with the kernels earlier than kernel-2.6.18-194.3.1.el5.

As an interruption mechanism used by peripheral component interconnect (PCI) devices, the message signaled interrupts (MSI) is applicable to NICs instead of a multi-core system. MSI-X is the enhanced version of MSI. Enabling MSI-X for an NIC driver improves the network performance, but increases the OS load. If the OS cannot adapt to the NIC running status or process data (especially for data with extremely large traffic), the OS becomes abnormal. After MSI-X is disabled, the NIC works in IO-APIC-level mode. In this way, the OS load is light, and the OS is normal.

"Message signaled interrupts (MSI) is an optional feature that enables PCI devices to request service by writing a system-specified message to a system-specified address (PCI DWORD memory write transaction). The transaction address specifies the message destination while the transaction data specifies the message. System software is expected to initialize the message destination and message during device configuration, allocating one or more non-shared messages to each MSI capable function."

2.  The solution provided in the RedHat website contains "Disable C-state in BIOS" (shown in red rectangular boxes in Figure 1). In addition, Broadcom5709 drivers are used for installing 64-bit Redhat5.4 in IBM X3652M2 and Dell R710; however, the incompatibility problem persists. edHat engineers reply that the driver of Broadcom NetXtreme II BCM5709 for RHEL 5.3/5.4 has a bug for managing Advanced Configuration and Power Interface (ACPI) power supplies. Therefore, when the NIC is working properly, the ACPI mistakes that the NIC is idle and disables the NIC.

For details about the IBM and Dell fault information, visit

http://sa.028life.com/?p=209#more-209

http://www.sealinger.com/archives/194

3.  Disable the ACPI power management system, that is, disable the C state. Disabling the P state ensures that the power management module disables Broadcom NetXtreme II BCM5709 correctly.

END