[T Series]An iSCSI Link-Down During Asianux 3 SP2IOmeter Read/Write Operations Led to an Not Responding Host

Publication Date:  2012-07-19 Views:  118 Downloads:  0
Issue Description

Product and version information:

  • This problem occurred on the S2600V1R5, S5500TV1R1, and S6800TV1R1, but the analysis indicated that the problem would occur on all storage systems.
    • The application server operating system was Asianux 3 SP2 for X86_64 with a native iSCSI initiator version iscsi-initiator-utils-6.2.0.868-0.18.1AXS3.

    Procedure:
    1.The storage array and the application server were connected through iSCSI links. A LUN was mapped to the Asianux host, and the a read/write test was performed by using the IOmeter.
    2.During the read/write operations, an iSCSI link-down occurred (caused by a cable removal or unexpected power-off). The host CPU utilization was close to 100% and the host was not responding even to SSH or KVM login attempts.

Alarm Information
None
Handling Process

1.Attempt to restore the iSCSI connections and wait 5 minutes for the host to start responding.
2.If the host is still not responding, restart the host.

Root Cause

1.Dynamo on Linux has an unlimited retry mechanism. When a slave block device returned an I/O error to upper-layer applications, Dynamo retried this failed I/O immediately, but the iSCSI driver returned this retry I/O as an error.
2.The above situation was a logical infinite loop. This infinite loop resulted in a CPU utilization of close to 100%.
The red rectangle in Figure1 shows the CPU utilization.
Figure 1
 

3.When the CPU utilization was close to 100%, the host may not respond to any external events.

 We ran the same test on a RedHat5.4 host, and the same symptoms occurred.

Suggestions
When using the IOmeter on a Linux host which is connected to the storage array through iSCSI connections, do not remove network cables and maintain a smooth network connection to avoid iSCSI link-downs. Otherwise, the Linux host CPU utilization may approach 100%.

END