No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionCloud 6.3.1.1 Troubleshooting Guide 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Abnormal Master and Slave Database Instances

Abnormal Master and Slave Database Instances

Symptom

The master and slave database instances are both abnormal, and physical database files are lost or damaged. This scenario involves the loss of MySQL, Redis software, or instance directory.

Fault Locating
  1. Use PuTTY to log in as the dbuser user to the active and standby nodes where database instance resides.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following commands to check whether the instance file is lost:

    cd /opt/

    ll

  3. Check whether the MySQL or Redis directory exists. If the directory does not exist, Troubleshooting step by step. If the directory exists, check whether files in the directory are damaged. If files are not damaged, skip the step for Initial Installation of Database Software, and Recreating the Master and Slave Database Instances directory.

Troubleshooting

Initial Installation of Database Software
Procedure
  1. Use PuTTY to log in to the om_core1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Install database software by recreating pod.

    Run the following command to obtain the name of the pod corresponding to the IP address of DBAgent. namespaces indicates the namespace where the faulty instance resides.

    kubectl get pod -n {namespaces} -owide| grep dbagent

    Information similar to the following is displayed:

    dbagent-fl17s     1/1       Running   0          19h       10.8.41.66   paas-10-8-41-66
    dbagent-ndcqh     1/1       Running   0          20h       10.8.41.73   paas-10-8-41-73

  3. Run the following command to delete the DBAgent services pod of the master and slave database.namespaces indicates the namespace where the faulty instance resides.

    kubectl delete pod dbagent-ndcqh -n {namespaces}

    Information similar to the following is displayed:

    pod "dbagent-ndcqh" deleted
    NOTE:
    • After the pod is deleted, the container re-establishes a new pod. If you want to confirm the pod, perform 2.
    • Preceding operations must be performed on the node where kubectl is deployed.

Recreating the Master and Slave Database Instances
  1. Use PuTTY to log in as the paas user to the master and slave database instances respectively.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run following commands to recreate the database:

    cd /opt/paas/oss/manager/apps/DBAgent/bin

    ./dbsvc_tool -cmd repair-db-node -type mysql

    Information similar to the following is displayed:

     success
    NOTE:
    • When recreating the database, the database process whose master and slave instance have been lost must be stopped. The master database must be re-created before the slave database.
    • After the instance is successfully created, perform logical recovery on the master database instance.
    • type indicates the database type to be restored. If the database is Redis, run the following command instead:

      ./dbsvc_tool -cmd repair-db-node -type redis

    • success indicates that the database instance is restored successfully.

  3. Restore master database instance on the active node of the database as instructed in section Restoring MySQL of Backup and Restoration Guide.

    NOTE:
    • The restoration only needs to be performed on the active node.
    • If the instance is not backed up before the disk is damaged, the data cannot be restored.
    • If you recreate a Redis database, the database table needs to be recreated.

Verification
  1. Use PuTTY to log in to the OM-Core01 node as the paas user.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following commands to check information about database:

    cd /opt/paas/oss/manager/apps/DBAgent/bin

    ./dbsvc_adm -cmd query-db-instance -type mysql

    Information similar to the following is displayed:

    DBInstanceId                             ClassId  Service Name               Region        Tenant Stage    IP          Port   State   DBType  Version  Role  Rpl Status   MasterID   
    apmdbsvr-10_90_73_163-3@10_90_73_164-3   primary  apmdbsvr-10_90_73_164-3    cn-global-1   om     Product 10.90.73.164 32082  Up      mysql   5.6.35   Master  Normal       apmdbsvr-10_90_73_163-3   
    apmdbsvr-10_90_73_178-21@10_90_73_179-21 primary  apmdbsvr-10_90_73_179-21   cn-global-1   om     Product 10.90.73.179 32080  Up      mysql   5.6.35   Slave Normal       apmdbsvr-10_90_73_178-21

    You only need to pay attention to the value of Rpl Status.

    If the statuses of Master and Slave are Normal, the database is running properly.

    If the statues are abnormal, contact technical support.

    NOTE:

    Check information about Redis database, run the following commands:

    ./dbsvc_adm -cmd query-db-instance -type redis

    You only need to pay attention to the value of Rpl Status.

    If the statuses of Master and Slave are Normal, the database is running properly.

    If the statues are abnormal, contact technical support.

Translation
Download
Updated: 2019-06-10

Document ID: EDOC1100063248

Views: 23295

Downloads: 37

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next