For the performance issue, the trouble shooting should start from Host, network and then storage array.
Generally speaking, most of performance issues will fixed by these four steps. But for this case, they are all normal but the performance is still low.
So we decide to proceed below steps:
1. Exclude the S5600's suspect, copy the data from windows host's local disks but the problem still exist.
2. Exclude the file system's suspect, create a new file system and export the share but the problem still exist.
3. Exclude the network, mount the file system by using the P0 port of the controller B. The read speed from local disks is 400MB/s and the write speed is 200MB/s. So we can confirm that the problem was caused by the network.
For the network issue, try below steps to trouble shooting:
Excluded the suspect of the cable, SFP, NIC, Port, Firewall and route configuration after these 6 steps.
The problem still exist after exchanged the HBA card, but it is normal when direct connect without switch. So the problem should be caused by switch.
After analyze the packets catched on the storage, we found the storage can answer the ARP message from switch. But there is no response from switch after that.
Everything become normal after customer replaced the switch, so it confirmed the problem was caused by the switch.