No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

HUAWEI CLOUD Stack 6.5.0 Troubleshooting Guide 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Abnormal Replication State of Slave Database Instance

Abnormal Replication State of Slave Database Instance

Symptom

Fault Symptom

The master database instance is normal, but the replication status of the slave database instance is abnormal.

Precautions
  • A password needs to be entered in interactive mode when running commands. (Directly entering a password has the password leakage risk.)
  • When the slave database instance is being restored, the Status of the slave database instance is on the RDBMS page of the FusionStage.
  • This section describes different operations on the Gauss database instances. Perform the operations as required.
Gauss Database Instance Fault Locating
  1. Use PuTTY to log in to the master node of the database instance as the dbuser user.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following commands to import environment variables of Gauss tools:

    cd /opt/gauss/app/bin

    . ~/appgsdb.bashrc

  3. Run the following command to query the status of the master database instance:

    gs_ctl query -D /opt/gauss/data/<Database instance ID>

    If information similar to the following is displayed, the master database instance is normal:

    Password: 
     Ha state:           
            LOCAL_ROLE                     : Primary
            STATIC_CONNECTIONS             : 1
            DB_STATE                       : Normal
            DETAIL_INFORMATION             : Normal
     Senders info:       
            SENDER_PID                     : 19505
            ...
            CHANNEL                        : 10.7.174.151:32083 -->10.7.174.152:59513

  4. Use PuTTY to log in to the master node of the database instance as the dbuser user.

    The default username is paas, and the default password is QAZ2wsx@123!.

  5. Perform 2 and 3 to query the status of the slave instance.

    If information similar to the following is displayed, the slave database instance is normal:
    Password: 
     Ha state:           
            LOCAL_ROLE                     : Standby
            STATIC_CONNECTIONS             : 0
            DB_STATE                       : NeedRepair
            DETAIL_INFORMATION             : Disconnected
     Senders info:       
            No information 
     Receiver info:      
            No information
    NOTE:
    • If the value of DB_STATE is NeedRepair, the replication between the master and slave database instances is abnormal.
    • DETAIL_INFORMATION indicates the cause of the exception.

(Method 1) Re-creating the Slave Database Instance on the GUI

Background

The console offers a graphical user interface (GUI) to re-create the slave database instance easily and quickly, greatly simplifying the operations in section (Method 3) Rebuild the Slave Database Instance.

Procedure
  1. Disable the failover function. For details, see section Disabling Database Failover.
  2. Use a browser to log in to the FusionStage OM zone console.

    1. Log in to ManageOne Maintenance Portal.
      • Login address: https://Address for accessing the homepage of ManageOne Maintenance Portal:31943, for example, https://oc.type.com:31943.
      • The default username is admin, and the default password is Huawei12#$.
    2. On the O&M Maps page, click the FusionStage link under Quick Links to go to the FusionStage OM zone console.

  3. Choose Application Development > Database > RDBMS from the main menu.
  4. Select the slave database instance with the abnormal replication state, and click Operation > Manual Repair to repair the slave database instance.
  5. After the database fault is rectified, enable the failover function.

(Method 2) Re-creating the Slave Database Instance in One-Click

Background

The slave database instance can be rebuilt in one-click mode by running a few commands, greatly simplifying the operations in section (Method 3) Rebuild the Slave Database Instance.

Procedure
  1. Disable the failover function. For details, see section Disabling Database Failover.
  2. Use PuTTY to log in to the manage_lb1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  3. Run the following command and enter the password of the root user to switch to the root user:

    su - root

    Default password: QAZ2wsx@123!

  4. Run the following command to query the node where the datamgmtservice service is deployed:

    kubectl get pod -nfst-manage -oyaml `kubectl get pod -nfst-manage | grep datamgmtservice | awk '{print $1}'` | grep hostIP

  5. Use PuTTY to log in to the node queried in 4.

    The default username is paas, and the default password is QAZ2wsx@123!.

  6. Run the following command to go to the installation directory of the dbsvc_adm tool:

    cd /opt/paas/oss/manager/apps/DBAgent/bin

  7. Run the following command to re-create the slave database instance:

    ./dbsvc_adm -cmd repair-db-instance -instid ossdbsvr-10_90_73_178-21@10_90_73_179-21 -slave ossdbsvr-10_90_73_179-21

    The following command output is displayed:

    Beginning repair db instance task.
    NOTE:

    Check whether a remote backup policy exists. If no remote backup policy exists, create a remote backup policy. For details about how to create a backup policy, see "Setting Backup Policies" in Backup and Restoration Guide.

  8. Use PuTTY to log in to the slave database node used for instance restoration as the paas user

    The default username is paas, and the default password is QAZ2wsx@123!.

  9. Run the following command to check the restoration results:

    tail -f /var/log/paas/oss/manager/DeployAgent/oss.dbrepair.trace |grep success

    The following information is displayed:

    2017-05-28 17:32:21.232(28646|140520325105408)[common:118]Slave replication status is ok, dbInstanceId:ossdbsvr-10_90_73_178-21@10_90_73_179-21, targetDC:ossdbsvr-10_90_73_179-21
    2017-05-28 17:32:21.233(28646|140520325105408)[common:118]Finished waiting for rebuilding the replication relationship of the slave database instance, dbInstanceId:wcptestsvr05171134-10_8_41_65-25@10_8_41_66-25, targetDC:ossdbsvr-10_90_73_179-21
    2017-05-28 17:32:21.233(28646|140520325105408)[common:118]repairSlave success, dbInstanceId:ossdbsvr-10_90_73_178-21@10_90_73_179-21, targetDC:ossdbsvr-10_90_73_179-21
    2017-05-28 17:32:21.254(28646|140520325105408)[proc:370]runCommand result=0

    If the message "repairSlave success" is displayed, the restoration succeeds. If the restoration fails, contact technical support.

  10. After the database fault is rectified, enable the failover function.

(Method 3) Rebuild the Slave Database Instance

Disabling Database Failover
Background

The failover function needs to be disabled before troubleshooting to prevent unexpected database failovers. After this function is disabled, the master/slave failover does not take place.

Procedure
  1. Use PuTTY to log in to the manage_lb1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following command and enter the password of the root user to switch to the root user:

    su - root

    Default password: QAZ2wsx@123!

  3. Run the following command to obtain the name of the pod corresponding to DBHASwitchService:

    kubectl get pod -n fst-manage | grep dbhaswitch | grep Running | awk '{ print $1 }'

    If the following command output is displayed, the failover function is successfully disabled:

    dbhaswitchservice-3302270813-1n452
    dbhaswitchservice-3302270813-cp154

  4. Run the following command to enter the pod corresponding to DBHASwitchService:

    kubectl exec dbhaswitchservice-3302270813-1n452 -n fst-manage -it sh

    NOTE:

    dbhaswitchservice-3302270813-1n452 indicates the name obtained in 3. If there are multiple names, set pod in the preceding command to any one of them.

  5. Run the following commands to go to the installation directory of the database failover tool:

    cd /opt/apps/DBHASwitchService/bin

  6. Run the following command to disable the failover function:

    ./switchtool.sh -cmd set-ignore-nodes -nodes 10_90_73_178,10_90_73_179

    If the following command output is displayed, the failover function is successfully disabled:

    Successful
    NOTE:

    -nodes: IDs of abnormal master and slave database nodes.

Restoring the Replication Status of a Slave Gauss Database Instance
Background

If management nodes are unavailable, you can follow the instructions provided in this section to restore the replication status of the slave Gauss database instance.

Procedure
  1. Use PuTTY to log in to the node where the slave instance resides as the paas user.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following command to go to the installation directory of the dbsvc_tool tool:

    cd /opt/paas/oss/manager/apps/DBAgent/bin/

  3. Run the following command to restore the replication status of the slave instance:

    ./dbsvc_tool -cmd repair-db-instance -method rebuild -instid apmdbsvr-10_90_73_178-21@10_90_73_179-21

    If success is displayed in the command output, the restoration is successful.

    success
    • If restoration is successful, no further action is required.
    • If restoration fails, perform 4.

  4. Contact technical support for assistance.

Verification

  1. Use PuTTY to log in to the manage_db1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following commands to check information about database:

    cd /opt/paas/oss/manager/apps/DBAgent/bin

    ./dbsvc_adm -cmd query-db-instance | grep gauss

    Information similar to the following is displayed:

    DBInstanceId                                  ClassId  InstNumber                    Tenant          IP             Port   State  DBType  Version            Role    Rpl Status  MasterID                      GuardMode  DataCheckSum  isSSL
    apmdbsvr-10_186_66_155-1@10_186_67_174-1      primary  apmdbsvr-10_186_66_155-1      fst-manage      10.186.66.155  32081  Up     gauss   V100R003C20SPC112  Master  Normal      --                            --         1520954391    off
    apmdbsvr-10_186_66_155-1@10_186_67_174-1      primary  apmdbsvr-10_186_67_174-1      fst-manage      10.186.67.174  32081  Up     gauss   V100R003C20SPC112  Slave   Normal      apmdbsvr-10_186_66_155-1      --         1520954391    off

    You only need to pay attention to the value of Rpl Status.

    If the statuses of Master and Slave are Normal, the database is running properly.

    If the statues are abnormal, contact technical support.

Translation
Download
Updated: 2019-06-01

Document ID: EDOC1100062375

Views: 1284

Downloads: 12

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next