1. Attempt to restore the iSCSI connections and wait 5 minutes for the host to start responding.
2. If the host is still not responding, restart the host.
Dynamo on Linux has an infinite retry mechanism. When a slave block device returned an i/o error to upper-layer applications, Dynamo retried this failed I/O immediately, but the iSCSI driver returned this retry I/O as an error. The above situation was a logical infinite loop. This infinite loop resulted in a CPU utilization of close to 100% as shown in Figure1.
Figure 1 CPU utilization
When the CPU utilization was close to 100%, the host may not respond to any external events.
We ran the same test on a Red Hat5.4 host, and the same symptoms occurred
When using IOmeter on a Linux host which is connected to the storage array through iSCSI connections, avoid iSCSI link-downs; otherwise, the Linux host CPU utilization may approach 100%.