No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search


To have a better experience, please upgrade your IE browser.


FusionCloud Troubleshooting Guide 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Operations Before Troubleshooting

Operations Before Troubleshooting


If you have located the cause of the database fault before troubleshooting, skip this chapter. If you need to analyze the cause of the fault after troubleshooting, perform the following operations to collect the required information.

  1. You are advised to perform emergency recovery during off-peak hours.
  2. During the restoration, do not perform operations that affect the database functions, for example, add, delete, modify the database and table, change the password and user, and start or stop the database or database node.


Database instance apmdbsvr-10_90_73_179 is used as an example in the following procedure:

  1. Use PuTTY to log in to the node where the abnormal database instance is deployed as the paas user.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following command to switch to the root user:

    su - root

  3. Run the following commands to back up the log and configuration files of the abnormal database instance:

    cp /opt/mysql/data/apmdbsvr-10_90_73_179/mysql.err /opt/mysql/data/apmdbsvr-10_90_73_179/mysql.err.bak

    cp /opt/mysql/data/apmdbsvr-10_90_73_179/auto.cnf /opt/mysql/data/apmdbsvr-10_90_73_179/auto.cnf.bak

  4. Run the following command to obtain the process ID of the abnormal database instance:

    ps -ef | grep apmdbsvr-10_90_73_179

    Information similar to the following is displayed:

    root     17878 25404  0 06:23 pts/5    00:00:00 grep --color=auto apmdbsvr
    dbuser   28409     1  0 Apr24 ?        00:00:00 /bin/sh /opt/mysql/bin/mysqld_safe --defaults-file=/opt/mysql/data/apmdbsvr-10_90_73_179-21/my.cnf --safe-user-create --skip-symbolic-links --ledir=/opt/mysql/bin --basedir=/opt/mysql --plugin-dir=/opt/mysql/lib/plugin
    dbuser   29796 28409  1 Apr24 ?        00:13:09 /opt/mysql/bin/mysqld --defaults-file=/opt/mysql/data/apmdbsvr-10_90_73_179-21/my.cnf --basedir=/opt/mysql --datadir=/opt/mysql/data/apmdbsvr-10_90_73_179-21 --plugin-dir=/opt/mysql/lib/plugin --safe-user-create --skip-symbolic-links --log-error=/opt/mysql/data/apmdbsvr-10_90_73_179-21/mysql.err --pid-file=/opt/mysql/data/apmdbsvr-10_90_73_179-21/ --socket=/opt/mysql/data/apmdbsvr-10_90_73_179-21/apmdbsvr-10_90_73_179-21.sock --port=32080

  5. Run the following commands to save the instance startup time to the specified file:

    ps -p 28409 -o lstart > apmdbsvr-10_90_73_179_mysqld_safe_time

    ps -p 29796 -o lstart > apmdbsvr-10_90_73_179_mysqld_time

    • 28409 is the process ID of the mysqld_safe instance obtained in 4.
    • 29796 is the process ID of the mysqld instance obtained in 4.

Fault Locating

  1. Use PuTTY to log in to the om_core1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following commands to check information about database:

    cd /opt/paas/oss/manager/apps/DBAgent/bin

    ./dbsvc_adm -cmd query-db-instance | grep mysql

    Information similar to the following is displayed:

    DBInstanceId                             ClassId  Service Name               Region        Tenant Stage    IP          Port   State   DBType  Version  Role  Rpl Status   MasterID   
    apmdbsvr-10_90_73_163-3@10_90_73_164-3   primary  apmdbsvr-10_90_73_164-3    cn-global-1   om     Product 32082  Up      mysql   5.6.35   Master  Normal       apmdbsvr-10_90_73_163-3   
    apmdbsvr-10_90_73_178-21@10_90_73_179-21 primary  apmdbsvr-10_90_73_179-21   cn-global-1   om     Product 32080  Up      mysql   5.6.35   Slave Abnormal (101)      apmdbsvr-10_90_73_178-21

    The command output varies depending on the version of the database service. Pay attention only to the value of Rpl Status.

    • Normal indicates that the replication status is normal.
    • Abnormal indicates abnormality, and the subsequent numerals in brackets is the status code.

    In this example, the replication status of the apmdbsvr-10_90_73_178-21@10_90_73_179-21 instance is abnormal. indicates the IP address of the node where the database resides and 32080 indicates the port number of the database instance. -- indicates a single-instance node. There is no replication status for a single node.

  3. If the Master and Slave statuses in the preceding 3 are Abnormal(101), perform Abnormal Master and Slave Database Instances.

    If the Master status is Normal and the Slave status is Abnormal(101), perform Abnormal Slave DatabaseInstances.

    For details about the status code, see section Database Instance Replication Status Is Abnormal in Troubleshooting Guide. If there is abnormal replication state of the slave database node instance, perform Abnormal Replication State of Slave Database Instance. Following methods for restoration are provided:

    • Re-establish the slave database node instance.
    • Rebuild the slave database node instance in one-click.
    • Rebuild the slave database node instance on the GUI.

Updated: 2019-06-10

Document ID: EDOC1100063248

Views: 22641

Downloads: 37

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Previous Next