No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

OceanStor 9000 V300R005C00 File System Feature Guide 11

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Troubleshooting

Troubleshooting

A Remote Replication Pair Fails to Be Created

The failure does not damage a file system but prevents an administrator from using remote replication to protect a file system.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Click Create. Select the local directory that you want to add to a remote replication pair, select a replication channel, specify a remote replication pair name, a remote directory, and other parameters, and click OK. Then, an error message is displayed.

Possible Causes

  • Possible cause 1: The rules for creating a remote replication pair are not met. For example, the specified remote replication pair name already exists, or the local or remote directory does not exist.
  • Possible cause 2: The CM process is abnormal.
  • Possible cause 3: The CCDB process is abnormal.
  • Possible cause 4: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-28  Troubleshooting flowchart

Procedure

  • Possible cause 1: The rules for creating a remote replication pair are not met.
    1. Modify parameters of the remote replication pair.

      • If Replication name is conflict is displayed, change the remote replication pair name.
      • If Directory is not exist is displayed, select an existing directory.

    2. Create a remote replication pair again and check whether it is created successfully.

  • Possible cause 2: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Create a remote replication pair again and check whether it is created successfully.

  • Possible cause 3: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Create a remote replication pair again and check whether it is created successfully.

  • Possible cause 4: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Create a remote replication pair again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Remote Replication Pair Fails to Be Deleted

The failure does not damage a file system but prevents an administrator from deleting an existing remote replication pair.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that you want to delete, click Delete, confirm the risk, and click OK. Then, error message Cannot perform this operation in the status or The communication is abnormal or the system is busy is displayed, and the remote replication pair fails to be deleted.

Possible Causes

  • Possible cause 1: The remote replication pair is not in the split state.
  • Possible cause 2: The CM process is abnormal.
  • Possible cause 3: The CCDB process is abnormal.
  • Possible cause 4: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-29  Troubleshooting flowchart

Procedure

  • Possible cause 1: The remote replication pair is not in the split state.
    1. If Cannot perform this operation in the status is displayed, check whether the remote replication pair is in the split state.

    2. Split the remote replication pair.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that you want to split, choose More > Split, confirm the risk, and click OK.

    3. After confirming that the remote replication pair is split, delete the pair again and check whether it is deleted successfully.

  • Possible cause 2: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Delete the remote replication pair again and check whether it is deleted successfully.

  • Possible cause 3: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Delete the remote replication pair again and check whether it is deleted successfully.

  • Possible cause 4: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Delete the remote replication pair again and check whether it is deleted successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Remote Replication Pairs Information Fails to Be Queried

The failure does not damage a file system but prevents an administrator from querying information about remote replication pairs.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Click Refresh. Then, an error message is displayed.

Possible Causes

  • Possible cause 1: The CM process is abnormal.
  • Possible cause 2: The CCDB process is abnormal.
  • Possible cause 3: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-30  Troubleshooting flowchart

Procedure

  • Possible cause 1: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Query information about remote replication pairs again and check whether the information is displayed successfully.

  • Possible cause 2: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Query information about remote replication pairs again and check whether the information is displayed successfully.

  • Possible cause 3: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Query information about remote replication pairs again and check whether the information is displayed successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Properties of a Remote Replication Pair Fails to Be Modified

The failure does not damage a file system but prevents an administrator from modifying properties of a remote replication pair.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair whose properties you want to modify, click Properties, modify the properties, and click OK. Then, error message The pair name already exists or The communication is abnormal or the system is busy is displayed, and the properties fail to be modified.

Possible Causes

  • Possible cause 1: The specified remote replication pair name already exists.
  • Possible cause 2: The CM process is abnormal.
  • Possible cause 3: The CCDB process is abnormal.
  • Possible cause 4: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-31  Troubleshooting flowchart

Procedure

  • Possible cause 1: The specified remote replication pair name already exists.
    1. If The pair name already exists is displayed, change the pair name.
    2. Modify properties of the remote replication pair again and check whether the properties are modified successfully.

  • Possible cause 2: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Modify properties of the remote replication pair again and check whether the properties are modified successfully.

  • Possible cause 3: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Modify properties of the remote replication pair again and check whether the properties are modified successfully.

  • Possible cause 4: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Modify properties of the remote replication pair again and check whether the properties are modified successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Data in a Remote Replication Pair Fails to Be Synchronized

The failure does not damage a file system but prevents an administrator from using remote replication to protect a file system.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that needs data synchronization, click Synchronize, confirm the risk, and click OK. Then, error message The operation cannot be performed due to the pair status or The communication is abnormal or the system is busy is displayed, and the remote replication pair fails to synchronize data.

Possible Causes

  • Possible cause 1: The remote replication pair is synchronizing data.
  • Possible cause 2: The link between the primary and secondary clusters is down.
  • Possible cause 3: The CM process is abnormal.
  • Possible cause 4: The CCDB process is abnormal.
  • Possible cause 5: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-32  Troubleshooting flowchart

Procedure

  • Possible cause 1: The remote replication pair is synchronizing data.
    1. If The operation cannot be performed due to the pair status is displayed, check whether the remote replication pair is in the Synchronizing state.

      • If yes, the remote replication pair is synchronizing data, and no further action is required.
      • If no, go to Possible cause 2.

  • Possible cause 3: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Enable the remote replication pair to synchronize data again and check whether data is synchronized successfully.

  • Possible cause 4: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Enable the remote replication pair to synchronize data again and check whether data is synchronized successfully.

  • Possible cause 5: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Enable the remote replication pair to synchronize data again and check whether data is synchronized successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Remote Replication Pair Fails to Be Split

The failure does not damage a file system but prevents an administrator from deleting a remote replication pair or initiating a primary/secondary switchover.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that you want to split, choose More > Split, confirm the risk, and click OK. Then, error message The operation cannot be performed due to the pair status or The communication is abnormal or the system is busy is displayed, and the remote replication pair fails to be split.

Possible Causes

  • Possible cause 1: The remote replication pair is in the split state.
  • Possible cause 2: The CM process is abnormal.
  • Possible cause 3: The CCDB process is abnormal.
  • Possible cause 4: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-33  Troubleshooting flowchart

Procedure

  • Possible cause 1: The remote replication pair is in the split state.
    1. If The operation cannot be performed due to the pair status is displayed, check whether the remote replication pair is in the split state.

      • If yes, the remote replication pair is split, and no further action is required.
      • If no, go to Possible cause 2.

  • Possible cause 2: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Split the remote replication pair again and check whether it is split successfully.

  • Possible cause 3: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Split the remote replication pair again and check whether it is split successfully.

  • Possible cause 4: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Split the remote replication pair again and check whether it is split successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Primary/Secondary Switchover Fails to Be Initiated

The failure does not damage a file system but prevents an administrator from using remote replication for service takeover.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair for which you want to initiate a primary/secondary switchover, choose More > Primary/Secondary Switchover, confirm the risk, and click OK. Then, error message Cannot perform this operation in the status, The secondary data of the remote replication is in the inconsistent state, The secondary directory of the pair is in read-only state, This operation cannot be performed because the secondary directory is in the rollback state, or The communication is abnormal or the system is busy is displayed, and the primary/secondary switchover fails.

Possible Causes

  • Possible cause 1: The remote replication pair is not in the split state.
  • Possible cause 2: Initial synchronization of the pair is not complete, causing data status inconsistency between the primary and secondary directories.
  • Possible cause 3: The secondary directory is read-only.
  • Possible cause 4: The snapshot of the secondary directory is in the rollback state.
  • Possible cause 5: The CM process is abnormal.
  • Possible cause 6: The CCDB process is abnormal.
  • Possible cause 7: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-34  Troubleshooting flowchart

Procedure

  • Possible cause 1: The remote replication pair is not in the split state.
    1. If Cannot perform this operation in the status is displayed, check whether the remote replication pair is in the split state.

    2. Split the remote replication pair.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that you want to split, choose More > Split, confirm the risk, and click OK.

    3. After confirming that the remote replication pair is split, initiate a primary/secondary switchover again and check whether the switchover is implemented successfully.

  • Possible cause 2: Initial synchronization of the pair is not complete, causing data status inconsistency between the primary and secondary directories.
    1. If The secondary data of the remote replication is in the inconsistent state is displayed, confirm with the administrator whether the pair has completed initial synchronization.

    2. Enable the remote replication pair to synchronize data.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that needs data synchronization, click Synchronize, confirm the risk, and click OK.

    3. Initiate a primary/secondary switchover for the remote replication pair again and check whether the switchover is implemented successfully.

  • Possible cause 3: The secondary directory is read-only.
    1. If The secondary directory of the pair is in read-only state is displayed, disable write protection for the secondary directory of the pair.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair, choose More > Disable Secondary Directory Write Protection, confirm the risk, and click OK.

    2. After confirming that Secondary Directory Write Protection of the pair is Read-write, initiate a primary/secondary switchover again and check whether the switchover is implemented successfully.

  • Possible cause 4: The snapshot of the secondary directory is in the rollback state.
    1. If This operation cannot be performed because the secondary directory is in the rollback state is displayed, log in to the CLI.

      Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run cli_start -u admin. Enter the password of user admin to log in to the CLI.

    2. Run show remote_replication general <id> and check whether the value of IsRoll is Yes.

      In the command, <id> indicates the ID of a remote replication pair. You can press Ctrl+A to view IDs of all pairs while using the command.

    3. After snapshot rollback of the secondary directory is complete (that is, when the value of IsRoll is No), initiate a primary/secondary switchover again and check whether the switchover is implemented successfully.

  • Possible cause 5: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Initiate a primary/secondary switchover for the remote replication pair again and check whether the switchover is implemented successfully.

  • Possible cause 6: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Initiate a primary/secondary switchover for the remote replication pair again and check whether the switchover is implemented successfully.

  • Possible cause 7: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Initiate a primary/secondary switchover for the remote replication pair again and check whether the switchover is implemented successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Secondary Directory Write Protection Fails to Be Disabled

The failure does not damage a file system but prevents a secondary directory from taking over upper-layer services and prevents an administrator from initiating a primary/secondary switchover.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair for which you want to disable secondary directory write protection, choose More > Disable Secondary Directory Write Protection, confirm the risk, and click OK. Then, error message Cannot perform this operation in the status, The secondary data of the remote replication is in the inconsistent state, This operation cannot be performed because the secondary directory is in the rollback state, or The communication is abnormal or the system is busy is displayed, and the secondary directory write protection fails to be disabled.

Possible Causes

  • Possible cause 1: The remote replication pair is not in the split or abnormal interruption state.
  • Possible cause 2: Initial synchronization of the pair is not complete, causing data status inconsistency between the primary and secondary directories.
  • Possible cause 3: The snapshot of the secondary directory is in the rollback state.
  • Possible cause 4: The CM process is abnormal.
  • Possible cause 5: The CCDB process is abnormal.
  • Possible cause 6: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-35  Troubleshooting flowchart

Procedure

  • Possible cause 1: The remote replication pair is not in the split or abnormal interruption state.
    1. If Cannot perform this operation in the status is displayed, check whether the remote replication pair is in the split state at the primary end, or check whether the remote replication pair is in the split or abnormal interruption state at the secondary end.

    2. Split the remote replication pair.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that you want to split, choose More > Split, confirm the risk, and click OK.

    3. After confirming that the remote replication pair is split, disable secondary directory write protection again and check whether it is disabled successfully.

  • Possible cause 2: Initial synchronization of the pair is not complete, causing data status inconsistency between the primary and secondary directories.
    1. If The secondary data of the remote replication is in the inconsistent state is displayed, confirm with the administrator whether the pair has completed initial synchronization.

    2. Enable the remote replication pair to complete initial synchronization.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair, click Synchronize, confirm the risk, and click OK.

    3. Disable secondary directory write protection again and check whether it is disabled successfully.

  • Possible cause 3: The snapshot of the secondary directory is in the rollback state.
    1. If This operation cannot be performed because the secondary directory is in the rollback state is displayed, log in to the CLI.

      Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run cli_start -u admin. Enter the password of user admin to log in to the CLI.

    2. Run show remote_replication general <id> and check whether the value of IsRoll is Yes.

      In the command, <id> indicates the ID of a remote replication pair. You can press Ctrl+A to view IDs of all pairs while using the command.

    3. After snapshot rollback of the secondary directory is complete (that is, when the value of IsRoll is No), disable secondary directory write protection again and check whether it is disabled successfully.

  • Possible cause 4: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Disable secondary directory write protection again and check whether it is disabled successfully.

  • Possible cause 5: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Disable secondary directory write protection again and check whether it is disabled successfully.

  • Possible cause 6: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Disable secondary directory write protection again and check whether it is disabled successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Secondary Directory Write Protection Fails to Be Enabled

The failure does not damage a file system but prevents an administrator from initiating data synchronization for a remote replication pair.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair for which you want to enable secondary directory write protection, choose More > Enable Secondary Directory Write Protection, confirm the risk, and click OK. Then, error message Cannot perform this operation in the status, The secondary data of the remote replication is in the inconsistent state, This operation cannot be performed because the secondary directory is in the rollback state, or The communication is abnormal or the system is busy is displayed, and the secondary directory write protection fails to be enabled.

Possible Causes

  • Possible cause 1: The remote replication pair is not in the split state.
  • Possible cause 2: Initial synchronization of the pair is not complete, causing data status inconsistency between the primary and secondary directories.
  • Possible cause 3: The snapshot of the secondary directory is in the rollback state.
  • Possible cause 4: The CM process is abnormal.
  • Possible cause 5: The CCDB process is abnormal.
  • Possible cause 6: The running status of the file system is abnormal.

Fault Diagnosis

Figure 6-36  Troubleshooting flowchart

Procedure

  • Possible cause 1: The remote replication pair is not in the split state.
    1. If Cannot perform this operation in the status is displayed, check whether the remote replication pair is in the split state.

    2. Split the remote replication pair.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that you want to split, choose More > Split, confirm the risk, and click OK.

    3. After confirming that the remote replication pair is split, enable secondary directory write protection again and check whether it is enabled successfully.

  • Possible cause 2: Initial synchronization of the pair is not complete, causing data status inconsistency between the primary and secondary directories.
    1. If The secondary data of the remote replication is in the inconsistent state is displayed, confirm with the administrator whether the pair has completed initial synchronization.

    2. Enable the remote replication pair to synchronize data.

      Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication. Select the remote replication pair that needs data synchronization, click Synchronize, confirm the risk, and click OK.

    3. Enable secondary directory write protection again and check whether it is enabled successfully.

  • Possible cause 3: The snapshot of the secondary directory is in the rollback state.
    1. If This operation cannot be performed because the secondary directory is in the rollback state is displayed, log in to the CLI.

      Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run cli_start -u admin. Enter the password of user admin to log in to the CLI.

    2. Run show remote_replication general <id> and check whether the value of IsRoll is Yes.

      In the command, <id> indicates the ID of a remote replication pair. You can press Ctrl+A to view IDs of all pairs while using the command.

    3. After snapshot rollback of the secondary directory is complete (that is, when the value of IsRoll is No), enable secondary directory write protection again and check whether it is enabled successfully.

  • Possible cause 4: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Enable secondary directory write protection again and check whether it is enabled successfully.

  • Possible cause 5: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Enable secondary directory write protection again and check whether it is enabled successfully.

  • Possible cause 6: The running status of the file system is abnormal.
    1. Clear the fault as instructed in The Running Status of a File System Fails to Be Queried or an Incorrect Running State Is Displayed.
    2. Enable secondary directory write protection again and check whether it is enabled successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Replication Zone Fails to Be Queried

The failure prevents an administrator from viewing information about a replication zone and node IP addresses in the replication zone.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Zone. Click Refresh. Then, an error message is displayed.

Possible Causes

  • Possible cause 1: The CM process is abnormal.
  • Possible cause 2: The CCDB process is abnormal.

Fault Diagnosis

Figure 6-37  Troubleshooting flowchart

Procedure

  • Possible cause 1: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Query the replication zone again and check whether the information is displayed successfully.

  • Possible cause 2: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

      • If yes, go to 2.
      • If no, contact technical support.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Query the replication zone again and check whether the information is displayed successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Properties of a Replication Zone Fail to Be Modified

The failure may cause some functions of a replication zone to be invalid. If the replication zone does not contain available IP addresses, no node participates in remote replication.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Zone. Select a replication zone whose properties you want to modify, click Properties, modify the properties, and click Apply. Then, error message The start IP address in the IP address segment is invalid, The end IP address in the IP address segment is invalid, The IP address segment is invalid, The excluded IP address is invalid, The replication zone is used by remote replication pairs or replication channels, or The replication zone is being used is displayed, and the properties fail to be modified.

Possible Causes

  • Possible cause 1: In the IP address segment, the start IP address is greater than the end IP address.
  • Possible cause 2: In the IP address segment, the start IP address, the end IP address, or the start and end IP addresses are not in the specified subnet segment.
  • Possible cause 3: The specified excluded IP address is invalid.
  • Possible cause 4: Remote replication pairs or replication channels are configured in the replication zone.
  • Possible cause 5: The replication zone is being used.
  • Possible cause 6: The CM process is abnormal.
  • Possible cause 7: The CCDB process is abnormal.

Fault Diagnosis

Figure 6-38  Troubleshooting flowchart

Procedure

  • Possible cause 1: In the IP address segment, the start IP address is greater than the end IP address.
    1. If The IP address segment is invalid is displayed, enter an IP address segment again and ensure that the end IP address is greater than or equal to the start IP address.
    2. Modify properties of the replication zone again and check whether the properties are modified successfully.

  • Possible cause 2: In the IP address segment, the start IP address, the end IP address, or the start and end IP addresses are not in the specified subnet segment.
    1. If The start IP address in the IP address segment is invalid or The end IP address in the IP address segment is invalid is displayed, enter an IP address segment again and ensure that the start and end IP addresses reside in the specified subnet segment.
    2. Modify properties of the replication zone again and check whether the properties are modified successfully.

  • Possible cause 3: The specified excluded IP address is invalid.
    1. If The excluded IP address is invalid is displayed, ensure that the excluded IP address resides in the specified subnet segment.
    2. Modify properties of the replication zone again and check whether the properties are modified successfully.

  • Possible cause 4: Remote replication pairs or replication channels are configured in the replication zone.
    1. If The replication zone is used by remote replication pairs or replication channels is displayed, add another IP address segment.

      When you modify an IP address segment, the preceding error message is displayed if remote replication pairs or replication channels are configured in the replication zone.

    2. After adding another IP address segment, delete the original IP address segment.
    3. Modify properties of the replication zone again and check whether the properties are modified successfully.

  • Possible cause 5: The replication zone is being used.
    1. If The replication zone is being used is displayed, log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Zone. Check whether any replication channel exists.

    2. After obtaining the administrator's approval, delete all replication channels.
    3. Modify properties of the replication zone again and check whether the properties are modified successfully.

  • Possible cause 6: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Modify properties of the replication zone again and check whether the properties are modified successfully.

  • Possible cause 7: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

      • If yes, go to 2.
      • If no, contact technical support.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Modify properties of the replication zone again and check whether the properties are modified successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Replication Channel Fails to Be Queried

The failure prevents an administrator from viewing information about replication channel.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Channel. Click Refresh. Then, an error message is displayed.

Possible Causes

  • Possible cause 1: The CM process is abnormal.
  • Possible cause 2: The CCDB process is abnormal.

Fault Diagnosis

Figure 6-39  Troubleshooting flowchart

Procedure

  • Possible cause 1: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Query the replication channel again and check whether the information is displayed successfully.

  • Possible cause 2: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

      • If yes, go to 2.
      • If no, contact technical support.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Query the replication channel again and check whether the information is displayed successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Replication Channel Fails to Be Created

The failure prevents remote replication pairs from being created. As a result, remote replication cannot be used.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Channel. Click Create and create a replication channel. Then, an error message is displayed, and the replication channel fails to be created.

Possible Causes

  • Possible cause 1: A replication channel to the same remote cluster has already been established.
  • Possible cause 2: The number of local or remote replication channels has reached the upper limit.
  • Possible cause 3: The local or remote replication zone does not contain available IP addresses.
  • Possible cause 4: The specified remote replication zone name does not exist.
  • Possible cause 5: The specified channel authentication IP address is unavailable.
  • Possible cause 6: The specified channel authentication password is incorrect.
  • Possible cause 7: The remote replication channel ID already exists.
  • Possible cause 8: The CM process is abnormal.
  • Possible cause 9: The CCDB process is abnormal.
  • Possible cause 10: The software version of the local cluster is different from that of the remote cluster.

Fault Diagnosis

Figure 6-40  Troubleshooting flowchart

Procedure

  • Possible cause 1: A replication channel to the same remote cluster has already been established.
    1. Check whether the error message is The replication channel has existed.

    2. On the replication channel management page, check whether a replication channel to the same remote cluster has already been established.

      • If yes, no further action is required because only one replication channel can be established between two clusters.
      • If no, contact technical support.

  • Possible cause 2: The number of local or remote replication channels has reached the upper limit.
    1. Check whether the error message is The number of local replication channels has reached the upper limit or The number of remote replication channels has reached the upper limit.

    2. Based on the specific error message, check whether the number of local or remote replication channels has reached the upper limit (32) on the replication channel management page.

      • If yes, go to 3.
      • If no, contact technical support.

    3. After obtaining the administrator's approval, delete unnecessary local or remote replication channels.
    4. Create a replication channel again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

  • Possible cause 3: The local or remote replication zone does not contain available IP addresses.
    1. Check whether the error message is No working replication node IP address exists in the local replication zone or No working replication node IP address exists in the remote replication zone.

    2. Based on the specific error message, log in to the local or remote DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Zone. Select the replication zone and check whether it contains available IP addresses.

      • If yes, contact technical support.
      • If no, go to 3.

    3. Modify the IP address segment that the replication zone contains to ensure that the front-end service IP address of at least one node is included.
    4. Create a replication channel again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

  • Possible cause 4: The specified remote replication zone name does not exist.
    1. Check whether the error message is The name of the remote replication zone does not exist.

    2. Correctly change the remote replication zone name.
    3. Create a replication channel again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

  • Possible cause 5: The specified channel authentication IP address is unavailable.
    1. Check whether the error message is The remote IP address that you enter cannot be the IP address in the local cluster or Failed to connect to the remote IP address.

    2. Based on the front-end service IP address of any node in the remote cluster, create a replication channel again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

  • Possible cause 6: The specified channel authentication password is incorrect.
    1. Check whether the error message is The password is incorrect.

    2. Use the correct password. Create a replication channel again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

  • Possible cause 7: The remote replication channel ID already exists.
    1. Check whether the error message is The remote replication channel ID already exists.

    2. Log in to the local DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Channel. Check whether a replication channel exists on the local DeviceManager but does not exist on the remote DeviceManager. Log in to the remote DeviceManager and repeat the preceding operations.

      • If yes, go to 3.
      • If no, contact technical support.

    3. On the local DeviceManager, delete the replication channels mentioned in 2.
    4. Create a replication channel again and check whether it is created successfully.

  • Possible cause 8: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Create a replication channel again and check whether it is created successfully.

  • Possible cause 9: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

      • If yes, go to 2.
      • If no, contact technical support.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Create a replication channel again and check whether it is created successfully.

  • Possible cause 10: The software version of the local cluster is different from that of the remote cluster.
    1. Contact the system administrator to check whether the cluster can be upgraded to the latest version.

      • If yes, go to 2.
      • If no, go to 3.

    2. Upgrade the cluster to the latest version and ensure that the software version of the local cluster is the same as that of the remote cluster. Then create a replication channel and check whether the replication channel is successfully created.

      • If yes, no further action is required.
      • If no, contact Huawei technical support.

    3. Ensure that the network is secure in the cluster of the latest version, and disable the replication link authentication by referring to the Testability and Maintainability MML Commands for Remote Replication. Create a replication channel again and check whether it is created successfully.

      • If yes, no further action is required.
      • If no, contact Huawei technical support.

Properties of a Replication Channel Fails to Be Modified

The failure may cause invalidity of the replication channel function or affect the performance of remote replication. As a result, remote replication pairs cannot be used for data synchronization.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Channel. Select a replication channel, click Properties, modify the properties, and click Apply. Then, an error message is displayed, and the properties of the replication channel fail to be modified.

Possible Causes

  • Possible cause 1: Property modification cannot be performed due to the status of the replication channel.
  • Possible cause 2: The CM process is abnormal.
  • Possible cause 3: The CCDB process is abnormal.

Fault Diagnosis

Figure 6-41  Troubleshooting flowchart

Procedure

  • Possible cause 1: Property modification cannot be performed due to the status of the replication channel.
    1. If The operation cannot be performed due to the replication channel status is displayed, check whether the running status of the replication channel is Link down.

    2. In the right navigation bar, click Alarm. Check whether alarm 0x400E0011 Replication Channel Is Disconnected about the replication channel is generated.

      • If yes, go to 3.
      • If no, contact technical support.

    3. Clear the alarm with the recommended actions in the alarm.
    4. Modify properties of the replication channel again and check whether the properties are modified successfully.

  • Possible cause 2: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Modify properties of the replication channel again and check whether the properties are modified successfully.

  • Possible cause 3: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

      • If yes, go to 2.
      • If no, contact technical support.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Modify properties of the replication channel again and check whether the properties are modified successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

A Replication Channel Fails to Be Deleted

The failure does not damage remote replication but prevents replication channel configuration from being deleted.

Symptom

Log in to DeviceManager. In the right navigation bar, choose Data Protection > Remote Replication > Replication Channel. Select a replication channel, click Delete, select I have read and understood consequences associated with performing this operation, and click OK. Then, an error message is displayed, and the replication channel fails to be deleted.

Possible Causes

  • Possible cause 1: The current cluster is not the initiator of the replication channel.
  • Possible cause 2: A remote replication pair is using the replication channel.
  • Possible cause 3: Deletion cannot be performed due to the status of the replication channel.
  • Possible cause 4: The CM process is abnormal.
  • Possible cause 5: The CCDB process is abnormal.

Fault Diagnosis

Figure 6-42  Troubleshooting flowchart

Procedure

  • Possible cause 1: The current cluster is not the initiator of the replication channel.
    1. Check whether the error message is The end is not the created end of the replication channel.

    2. On the property page of the replication channel, check whether Originating End is Yes.

      • If yes, contact technical support.
      • If no, go to 3.

    3. Log in to DeviceManager of the replication channel initiator. Delete the replication channel and check whether it can be deleted successfully.

  • Possible cause 2: A remote replication pair is using the replication channel.
    1. Check whether the error message is The replication channel is used by remote replication pairs.

    2. Choose Data Protection > Remote Replication > Remote Replication Pair. Check whether a pair is using the replication channel.

      • If yes, go to 3.
      • If no, contact technical support.

    3. After obtaining the administrator's approval, delete the pair that is using the replication channel.
    4. Delete the replication channel again and check whether it is deleted successfully.

  • Possible cause 3: Deletion cannot be performed due to the status of the replication channel.
    1. Check whether the error message is The operation cannot be performed due to the replication channel status.

  • Possible cause 4: The CM process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep snas_cm to check whether the CM process is abnormal.

    2. Restart the CM process.

      In the CLI, run kill -9 `ps -ef|grep snas_cm|grep -v grep|awk '{print $2}'` to restart the CM process.

    3. Delete the replication channel again and check whether it is deleted successfully.

  • Possible cause 5: The CCDB process is abnormal.
    1. Use PuTTY to log in to the storage system as user omuser based on the management IP address of the OceanStor 9000. Run su. Enter the login password of user root to switch to user root. In the CLI, run ps -ef |grep ccdb_server to check whether the CCDB process is abnormal.

      • If yes, go to 2.
      • If no, contact technical support.

    2. Restart the ccdb_server process.

      1. Run the following commands to stop the CCDB process:

        /opt/huawei/deploy/bin/daemon -s /opt/huawei/snas_cluster/bin/ccdb_server

      2. Run the following commands to restart the CCDB process:

        /opt/huawei/deploy/bin/daemon /opt/huawei/snas_cluster/bin/ccdb_server

    3. Delete the replication channel again and check whether it is deleted successfully.

      • If yes, no further action is required.
      • If no, contact technical support.

Translation
Download
Updated: 2019-03-30

Document ID: EDOC1000101823

Views: 19796

Downloads: 99

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next