No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionStorage V100R006C10 Block Storage Service Troubleshooting Guide 04

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
A Single FSM VM Fails (Deployed on a FusionSphere OpenStack Host)

A Single FSM VM Fails (Deployed on a FusionSphere OpenStack Host)

Symptom

When two FusionStorage Manager (FSM) VMs are deployed in active/standby mode and one FSM VM becomes faulty and cannot be restored after a restart, perform the operations provided in this section to rectify faults and quickly restore services.

This section applies to the FusionSphere OpenStack and FusionSphere OpenStack + FusionCompute scenarios in which FSM VMs are deployed on FusionSphere OpenStack hosts.

Two restoration plans are available in such cases:
  • Restore the faulty VM on its own host.
  • Restore the faulty VM on another host. This plan can be used if the host of the faulty VM fails and cannot be restored.

Possible Causes

The file system on the FSM VM is damaged.

Fault Diagnosis

None

Procedure

    Log in to the faulty VM using Virtual Network Computing (VNC).

    1. Log in to the FusionSphere OpenStack web client over the External API network.

      The login address is in the format of https://Reverse proxy IP address of FusionSphere OpenStack:8890. For example, https://192.168.211.90:8890.

      The default username is admin. The default password is FusionSphere123.

    2. Choose Cloud Service > FusionStorage.
    3. In the right pane, log in to the faulty FSM VM using VNC as user dsware.

      The default password of user dsware is IaaS@OS-CLOUD9!.

    1. Check whether the VNC login is successful.

      • If yes, go to 5.
      • If no, the host of the VM fails. Go to 8.

    Enable the high availability (HA) service.

    1. Run the following command and enter the password of user root to switch to user root:

      su - root

    2. Run the following command to check the HA status of the FSM VM:

      sh /opt/dsware/manager/setup/forCommonServer/checkFSMStatus.sh

      If the command fails to execute, the HA service encounters a fault. Reinstall the FSM VM and go to 8.

      Information similar to the following is displayed (the node whose HA_active value is active is the active node):

      Ha mode
      double 
      
      NodeName                                 HostName                                 HaVersion                StartTime                HA_active            HA allResOK          HARunPhase          
      DSM01                                    FSM01                                    V100R001C01              2015-10-30 16:02:04      active               normal               Actived             
      DSM02                                    FSM02                                    V100R001C01              2015-10-30 16:02:04      standby              normal               Deactived           
      
      NodeName                                 ResName                                  ResStatus                ResHAStatus              ResType             
      DSM01                                    DSMExternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMInternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMLocalExternalMgrIp                    Normal                   Normal                   Double_active       
      DSM01                                    DSMLocalInternalMgrIp                    Normal                   Normal                   Double_active       
      ......

    3. Check whether the HA allResOK values are normal.

    Query the HA status.

    1. Log in to the other FSM VM using VNC as user dsware.

      The default password of user dsware is IaaS@OS-CLOUD9!.

      If the login fails, contact technical support.

    2. Run the following command to switch to user root:

      The default password of user root is IaaS@OS-CLOUD8!.

      su - root

    3. Run the following command to check the HA status of the FSM VM:

      sh /opt/dsware/manager/setup/forCommonServer/checkFSMStatus.sh

      Information similar to the following is displayed:

      Ha mode
      double 
      
      NodeName                                 HostName                                 HaVersion                StartTime                HA_active            HA allResOK          HARunPhase          
      DSM01                                    FSM01                                    V100R001C01              2015-10-30 16:02:04      active               normal               Actived             
      DSM02                                    --                                       --                       --                       --                   --                   --                  
      
      NodeName                                 ResName                                  ResStatus                ResHAStatus              ResType             
      DSM01                                    DSMExternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMInternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMLocalExternalMgrIp                    Normal                   Normal                   Double_active       
      DSM01                                    DSMLocalInternalMgrIp                    Normal                   Normal                   Double_active       
      ......

    4. Check whether the HA_active column contains the active value.

    Complete pre-processing before data rebuilding.

    1. Run the following command to switch to the /opt/omm/oms/ directory:

      cd /opt/omm/oms/

    2. Run the following command to query the workspace and workspace_install soft links:

      ll

    3. Check whether the workspace and workspace_install soft links are correct.

      • If yes, go to 23.
      • If no, go to 15.

      The soft links are correct if the workspace soft link points to workspace0 and the workspace_install soft link points to workspace1.

      If information similar to the following is displayed, the workspace soft link is correct:

      ...
      lrwxrwxrwx  1 omm    omm    28 Apr 12 10:30 tools -> /opt/omm/oms/workspace/tools
      lrwxrwxrwx  1 root   root   10 Apr 13 16:00 workspace -> workspace0
      drwxr-x--- 16 omm    omm  4096 Apr 13 00:00 workspace0
      drwxr-x--- 15 omm    omm  4096 Apr 12 10:31 workspace1
      lrwxrwxrwx  1 root   root   10 Apr 13 16:00 workspace_install -> workspace1

    4. Run the following command to switch to the /opt/dsware/manager/setup directory:

      cd /opt/dsware/manager/setup

    5. Run the following command to stop the HA service:

      ./MonitorTool.sh stop

      NOTE:
      You can run the ps -ef | grep ha.bin command to check whether the HA process has been stopped.

    6. Run the following command to switch to the /opt/omm/oms/ directory:

      cd /opt/omm/oms/

    7. Run the following commands to switch the OMS workspace:

      mv workspace1 workspace_work

      mv workspace0 workspace1

      mv workspace_work workspace0

      rm workspace

      ln -s workspace0 workspace

      rm workspace_install

      ln -s workspace1 workspace_install

    8. Run the following command to query the workspace soft link:

      ll

      Ensure that the workspace soft link points to workspace0 and the workspace_install soft link points to workspace1.

    9. Run the following command to switch to the /opt/dsware/manager/setup directory:

      cd /opt/dsware/manager/setup

    10. Run the following command to start the HA service:

      ./MonitorTool.sh start

      NOTE:
      You can run the ps -ef | grep ha.bin command to check whether the HA process has been started.

    11. Run the following command to check the HA status of the FSM VM:

      sh /opt/dsware/manager/setup/forCommonServer/checkFSMStatus.sh

      Information similar to the following is displayed:

      Ha mode
      double 
      
      NodeName                                 HostName                                 HaVersion                StartTime                HA_active            HA allResOK          HARunPhase          
      DSM01                                    FSM01                                    V100R001C01              2015-10-30 16:02:04      active               normal               Actived             
      DSM02                                    --                                       --                       --                       --                   --                   --                  
      
      NodeName                                 ResName                                  ResStatus                ResHAStatus              ResType             
      DSM01                                    DSMExternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMInternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMLocalExternalMgrIp                    Normal                   Normal                   Double_active       
      DSM01                                    DSMLocalInternalMgrIp                    Normal                   Normal                   Double_active       
      ......

    Select a restoration plan.

    1. Select the restoration plan based on the host status.

      • If the host accommodating the faulty VM is running properly, go to 26 to restore the VM on this host.
      • If the host accommodating the faulty VM fails and cannot be restored, go to 38.

    Restore the VM on its own host.

    1. Log in to a controller host in the FusionSphere OpenStack system.
    2. Import environment variables. For details, see Importing Environment Variables in the FusionStorage Block Storage Service Software Installation Guide.
    3. Run the following command and take note of the ID of the faulty VM:

      nova list --all-t

    4. Run the following command to switch to repo directory:

      cd /opt/fusionplatform/data/fusionsphere/repo/

    5. Run the following command to download the FSM image file to repo directory:

      URL=$(cat /etc/init.cfg |grep -w repo |awk -F "//" '{print $NF }');curl -O -s -k https://{$URL}3rdvms/image/fusionstorage-manager.iso||curl -O -s -k http://{$URL}3rdvms/image/fusionstorage-manager.iso

      Example: curl -O -s -k http://172.28.10.101:8007/v1/AUTH_test/ZyppSrcPrj/3rdvms/image/fusionstorage-manager.iso

    6. Run the following command to calculate the md5 value of the image file:

      md5sum /opt/fusionplatform/data/fusionsphere/repo/fusionstorage-manager.iso

      Information similar to the following is displayed:
      d8f49f11bd67e2905f04766cdb76c88b  /opt/fusionplatform/data/fusionsphere/repo/fusionstorage-manager.iso

    7. Run the following command to import environment variables:

      export OS_IMAGE_API_VERSION=1

    8. Run the following command to query the ID of the FSM image file from Glance:

      glance image-list --all-t --name "fusionstoragemanager_image"

      Information similar to the following is displayed:
      +--------------------------------------+----------------------------+-------------+------------------+-----------+--------+
      | ID                                   | Name                       | Disk Format | Container Format | Size      | Status |
      +--------------------------------------+----------------------------+-------------+------------------+-----------+--------+
      | 7e34b03d-bc6e-4de4-8eae-b40257c83ab2 | fusionstoragemanager_image | iso         | bare             | 849750016 | active |
      +--------------------------------------+----------------------------+-------------+------------------+-----------+--------+
      
      In the command output, the value in the ID column specifies the image ID.

    9. Run the following command to query the md5 check value from Glance:

      glance image-show ID

      ID is obtained from 31.

      Information similar to the following is displayed:
      +------------------+--------------------------------------+
      | Property         | Value                                |
      +------------------+--------------------------------------+
      | checksum         | 5a27341931024168acc479073d96fb0e     |
      | container_format | bare                                 |
      | created_at       | 2016-08-24T10:56:39.694832           |
      | deleted          | False                                |
      | disk_format      | iso                                  |
      | id               | 7e34b03d-bc6e-4de4-8eae-b40257c83ab2 |
      | is_public        | False                                |
      | min_disk         | 0                                    |
      | min_ram          | 0                                    |
      | name             | fusionstoragemanager_image           |
      | owner            | 3a91eddee1a04cd08fe384e4f653db62     |
      | protected        | False                                |
      | size             | 849750016                            |
      | status           | active                               |
      | updated_at       | 2016-08-24T10:56:58.214174           |
      +------------------+--------------------------------------+
      
      In the command output, the value in the checksum row specifies the md5 check value.

    10. Compare the md5 values obtain from 29 and 32 and check whether they are consistent.

      • If yes, go to 36.
      • If no, go to 34.

    11. Run the following command to delete the FSM image instance from Glance:

      glance image-delete ID

      ID is obtained from 31.

    12. Run the following command to create a new FSM image instance and take note of the returned image ID:

      glance image-create --name fusionstoragemanager_image --disk-format iso --container-format bare --progress --file /opt/fusionplatform/data/fusionsphere/repo/fusionstorage-manager.iso

      Information similar to the following is displayed:
      [=============================>] 100%
      +------------------+--------------------------------------+
      | Property         | Value                                |
      +------------------+--------------------------------------+
      | checksum         | d8f49f11bd67e2905f04766cdb76c88b     |
      | container_format | bare                                 |
      | created_at       | 2016-08-25T02:19:16.939602           |
      | deleted          | False                                |
      | deleted_at       | None                                 |
      | disk_format      | iso                                  |
      | id               | 129437d2-c637-4c29-8c31-5783abd1356e |
      | is_public        | False                                |
      | min_disk         | 0                                    |
      | min_ram          | 0                                    |
      | name             | fusionstoragemanager_image           |
      | owner            | e321d42e981745ebaed4cf669fea9741     |
      | protected        | False                                |
      | size             | 849747968                            |
      | status           | active                               |
      | updated_at       | 2016-08-25T02:19:30.464424           |
      | virtual_size     | None                                 |
      +------------------+--------------------------------------+
      
      In the command output, 100% specifies that the image instance is successfully created, and the value in the id row specifies the image ID.

    13. Run the following command to restore the FSM VM based on the obtained VM ID and image ID:

      nova rebuild VM ID ISO image ID

      For example, run the following command:

      nova rebuild 779018b0-8713-4a40-b43a-e2700e7305b9 129437d2-c637-4c29-8c31-5783abd1356e

      You can use the nova list command to query the VM restoration status. If the Status value changes to Active and the Power State value changes to Running, the VM is restored successfully.

      After successfully restored, the VM automatically installs FSM software packages, completes configuration, and synchronizes data with the other FSM VM. The entire process takes about 35 minutes.

    14. After 35 minutes, log in to the FSM VM using VNC and run the following command:

      sh /opt/dsware/manager/setup/forCommonServer/checkFSMStatus.sh

      The VM is successfully restored if information similar to the following is displayed (the node whose HA_active value is active is the active node):

      Ha mode
      double 
      NodeName                                 HostName                                 HaVersion                StartTime                HA_active            HA allResOK          HARunPhase          
      DSM01                                    FSM01                                    V100R001C01              2015-10-30 16:02:04      active               normal               Actived             
      DSM02                                    FSM02                                    V100R001C01              2015-10-30 16:02:04      standby              normal               Deactived           
      
      NodeName                                 ResName                                  ResStatus                ResHAStatus              ResType             
      DSM01                                    DSMExternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMInternalMgrFloatIp                    Normal                   Normal                   Single_active       
      DSM01                                    DSMLocalExternalMgrIp                    Normal                   Normal                   Double_active       
      DSM01                                    DSMLocalInternalMgrIp                    Normal                   Normal                   Double_active       
      ......

      No further action is required.

    Delete and reinstall the FSM VM.

    1. Run the following command to switch to user dsware:

      su - dsware

    2. Run the following command to create a management data backup task and take note of the task ID:

      NOTE:

      Since the system has been hardened, you need to enter the username and password for login authentication after running the dswareTool command of FusionStorage Block. The default username is cmdadmin, and its default password is IaaS@PORTAL-CLOUD9!.

      The system supports authentication using environment variables so that you do not need to repeatedly enter the username and password for authentication each time you run the dswareTool command. For details, see Authentication Using Environment Variables in the FusionStorage Block Storage Service Software Installation Guide.

      sh /opt/dsware/client/bin/dswareTool.sh --op executeBackupCmd

      Enter username cmdadmin and its password as prompted. The default password is IaaS@PORTAL-CLOUD9!.

      The task is successfully created if information similar to the following is displayed:
      Operation finish successfully. Result Code:0
      task id: 20

    3. Run the following command to use the task ID obtained in 39 to query the task progress:

      sh /opt/dsware/client/bin/dswareTool.sh --op queryTaskInfo -id ID

    4. After the backup task is complete, copy the backup data to the local device or a third-party server.
    5. Log in to the FusionSphere OpenStack web client and delete the FSM VM.
    6. Reinstall the FSM VM and restore the VM data using the data backup. For details, see the Emergency Handling for Active and Standby FSM Node Faults in the FusionStorage Block Storage Service Emergency Handling Guide package.

      NOTE:
      The parameter values set for the reinstalled FSM VM must be the same as those set for the original FSM VM.

Related Information

None

Translation
Download
Updated: 2019-02-01

Document ID: EDOC1000175245

Views: 3186

Downloads: 4

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next