SFP module of S5500T V2 device doesn’t work after re-inserting frequently

Publication Date:  2015-07-24 Views:  177 Downloads:  0
Issue Description

After inserting the SFP module in the FC port, the indicator is off and the port can be used, the picture is as below:

Alarm Information
In the “event” logs, we can see the alarms as below:
1. The FC host port (Engine ENG0, controller B, port number H3) is disconnected
2. The replication link (link ID 256, local controller 0B, local port ENG0.B1.H3, remote controller 0B, remote port ENG0.B1.H2, remote device name SEMS5500T, serial number xxxxxxxxxxxxxxxxxxxx) was disconnected. Therefore, the remote devices cannot be accessed.
Handling Process
1. Make sure that there are redundant links between HOSTs and storage, then we did the test to exchange the SFP module between the normal FC port and problematic FC port, we found that the problem is related with the FC port, not SFP module;
2. Collect “operating data” and “system log” on the device manage portal;
3. In the “operating data” file, about FC port, we found that the SFP module can be recognized, but the status is offline, we can see the hardware can be discovered and there may be some problem about the software function;

4. According to the alarm time or the occurring time of the problem, we can check the “messages” logs, and search the key word “SFP” and “fibre module”, then we found the information as below:

5. With the key information above, we can see the SFP was removed at 10:08:11 physically, and the logical status became “link-down” 6 seconds later at 10:08:17, during the period, system started the checking of the speed about the SFP module at 10:08:14, the mismatch speed and link-up status appeared at the same time, then our system disabled the FC port.
Root Cause
The changing time of the logical status of the FC port is delayed, occasionally the system is checking the speed of the port at that time and finds the mismatch speed, and then the problem that the FC port is disabled with low probability happens.
The command to enable the FC port manually is not developed for current version V200R002C00, so we have to reset the controller to restore the FC port, when both controllers are normal, we can reset it one by one; for the permanent solution, we can upgrade the system software to V200R002C20SPC200 or higher
In the logs, we can see there are lots of records about re-inserting the SFP module with high frequency, and then the problem happened occasionally, so when the device is running normally, we suggest not re-inserting SFP modules with high frequency.