No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

LUN disconnected from server

Publication Date:  2015-01-02 Views:  78 Downloads:  0
Issue Description
Setup Details: - Mode: S5500T    Version: V200R001 & RedHat cluster server for MYSQL database of 800GB.

LUN are disconnected from the application server automatically,

on analysis we found that Customer is using RedHat cluster server for MYSQL database of 800GB, that need 20000 IOPS,
we found below observation
1. The current configuration of the box is designed to handle max 2000 IOPS.
2. Performance parameter were not properly configured,
3. Duplicate target settings found in one node of the application cluster
4. Multipathing configuration (we recommend Huawei Ultrapath)
hence performance issue occurred, which lead to disconnection of the Storage LUNS from the Cluster systems
Alarm Information
LUN disconnection hisotory

 First Time
  3:36:26 disconnected      0:00:17
  3:36:43 connected
Second Time
  22:19:34 disconnected   0:06:10
  22:25:44 connected
Third Time
  22:34:56 disconnected    0:06:39
  22:41:35 connected
Fourth Time
  22:55:13 disconnected    0:06:25
  23:01:38 connected
Fifth Time
  14:41:43 disconnected     0:06:16
  14:47:59 connected


Logs from the S5500T storage controllers
0x200e02120010||2013-09-12 15:33:22 UTC+08:00||The user (user-name admin) logged in to the device by using the source (172.29.0.100) successfully.
0x1201f8002d||2013-09-12 05:54:13 UTC+08:00||LUN copy(LUNCopy name 0, LUNCopy id 0, Source lun 0) has completed successfully
0x1201f9001e||2013-09-12 03:56:05 UTC+08:00||LUN (lun-id:4) was formatted successfully.
0x1201f9001e||2013-09-12 00:09:15 UTC+08:00||LUN (lun-id:5) was formatted successfully.
0x1201f80018||2013-09-11 20:36:57 UTC+08:00||LUN copy (luncopy-name:clone_81_131_Lun_1, luncopy-id:0, source-lun-id:0) started to be executed.
0x200e01f80010||2013-09-11 20:36:57 UTC+08:00||admin:172.29.0.206 started the LUN copy (LUN-copy-ID 0) successfully.
0x200e01f80001||2013-09-11 20:34:39 UTC+08:00||admin:172.29.0.206 created the LUN copy (LUN-copy-name clone_81_131_Lun_1) successfully.
0x1201f80015||2013-09-11 20:34:39 UTC+08:00||LUNCopy (luncopy-name:clone_81_131_Lun_1, luncopy-id:0, source-lun-id:0) is created successfully.
0x1201f80014||2013-09-11 20:34:39 UTC+08:00||Creating LUNCopy (luncopy-name:clone_81_131_Lun_1) started.
0x200e01f70001||2013-09-11 20:19:40 UTC+08:00||admin:172.29.0.206 created the LUN (owing-RAID-group-ID 1, LUN-capacity 838860800 KB, stripe-depth 64 KB) successfully.
0x1201f9001c||2013-09-11 20:19:40 UTC+08:00||Formatting LUN (lun-id:5) started.
0x12020e007e||2013-09-11 20:19:40 UTC+08:00||Creating LUN (lun-id:5) succeeded.
0x200e01f70001||2013-09-11 20:18:53 UTC+08:00||admin:172.29.0.206 created the LUN (owing-RAID-group-ID 1, LUN-capacity 838860800 KB, stripe-depth 64 KB) successfully.
0x1201f9001c||2013-09-11 20:18:53 UTC+08:00||Formatting LUN (lun-id:4) started.
0x12020e007e||2013-09-11 20:18:53 UTC+08:00||Creating LUN (lun-id:4) succeeded.
Handling Process
1. Collect the Logs from ISM tool and host side
2. Analyze the logs:-  find the operations ( storage side) that require IOPS, analyze the Target settings at Host side, Analyze the activity at host side that required IOPS.
3. Set the performance parameters and fine tune the settings to get max output from the storage box issue.
4. Delete old initiators and rescan for targets
5. Multipathing settings (we recommend the ultrapath)
Root Cause
1. The storage system has low IOPS capability, customer application environment has more IOPS requirement,
2. Storage Performance parameters were not configured properly,
3. Multipathing not configured properly 
Solution
Resolution steps need to perform at storage side
1: All of the LUN are currently mirrored with write-back;/ / suggested to set write-back, no mirror to increase the system performance
2: A total of four mapped LUN, // divided the LUN load with two controllers
• Two LUN ownership in the A controller
• Two LUN ownership in the B controller;
3: Customers did full amount LUN copies,but Currently resource pool doest not have capacity;/ /  recommonded to configure resource pool for snapshot and incremental LUN-Copy.

Resolution steps need to perform at host side
1: There are some errors on the host configuration:
• store IP on the host side has saved two different target device name , one of which is invalid, causing the host to repeatedly connect but the connection fails , you need to clear the
• / / one of the target is invalid, need to be cleared.
• Released on the ISM LUN mapping / / relieve map
• Unplug the iscsi port connection / / remove iscsi cable
• The host side execution iscsiadm-m node-p 172.29.64.152/153-u / / execute
• Then iscsiadm-m node view , you should see the name of a stored 20000:172.29.64.152153 and 172.29.64.152/153 ending two , these two need to delete / / check and remove these two
• The Executive iscsiadm-m node-o delete-p 172.29.64.152/153 deletes the configuration and confirm been deleted / / delete configuration
• Rediscover the target device iscsiadm-m discovery-t st-p 172.29.64.152/153 / / rescan
• Log iscsiadm-m node-p 172.29.64.152/153-l / / log in
• Connecting iscsi port connection / / connect iscsi cable
• Re-add the LUN mapping / / map LUN

Multipathing configuration
2 : The current multi-path using multi-path comes RedHat6.2 device mapper multipath, in the butt when our store , you need to add in the configuration file stores information about our Hurawei UltraPath software , otherwise , there may be some unpredictable problems
Need to check the / etc / multipath.conf devices in the Division I field whether the stored configuration , if not then you need to add
Add the following :
#
# HUAWEI: S5500T IALUA
#
device {
                vendor "HUAWEI | HUASY"
                product "S5500T | S3900 (*)"
                path_grouping_policy group_by_prio
                getuid_callout "/ lib / udev / scsi_id - whitelisted - device = / dev /% n"
                path_checker tur
               prio alua
                path_selector "round-robin 0"
                failback immediate
}
Reconfirmation blacklist {} configuration item is not set to the array shielding.
Finally, execute the following command to validate the configuration
multipath-F
service multipathd reload
multipath-v2

In addition, you must also open ALUA arrays side configuration. Operation: In the ISM interface, the corresponding startup configuration ALUA. / / initiate ALUA on corresponding initiator.
Suggestions
1. Check the storage box capability and masure how much max IOPS it can handle.
2. Consider the performance parameters, ( Cache, Controller ownership, channel, Multipathing, Disk level, Raid Level) 
3. chec host settings, some time duplicate target settings can impact the server performance
4. Multipathing configuration (we recommond huawei Ultrapath )

END