No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

HUAWEI CLOUD Stack 6.5.0 Troubleshooting Guide 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
etcd Restart Due to Inconsistent Data

etcd Restart Due to Inconsistent Data

Symptom

  • The following log information indicates that etcd restarts repeatedly:
    2018-05-15 16:00:45.985584 C | etcdmain: database file (/var/etcd-data/etcd-event/etcd-event-2/member/snap/db index 16737187) does not match with snapshot (index 21909430).
  • Alternatively, etcd restarts repeatedly because a panic log similar to the following is generated, which is caused by bbolt:
    panic: xxx
    xxx/github.com/coreos/bbolt/xxx

Possible Causes

etcd data is damaged.

Troubleshooting Method

  1. Use PuTTY to log in to the manage_lb1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following command and enter the password of the root user to switch to the root user:

    su - root

    Default password: QAZ2wsx@123!

  3. Run the following command to check whether all etcd instances are in the Running state (etcd-network is used as an example).

    kubectl -n fst-manage get pod|grep etcd|grep -v cse

    etcd-network-server-paas-10-118-29-153       1/1       Running          0          1h
    etcd-network-server-paas-10-118-29-169       1/1       Running          0          1h
    etcd-network-server-paas-10-118-29-73        1/1       Error            0          1h

    The preceding command output shows that etcd-network-server-paas-10-118-29-73 is abnormal.

  4. Use PuTTY to log in to the node using the IP address of the abnormal node.
  5. Run the following command to query logs of the critical level in run log etcd-event.log in the directory:

    vi /var/paas/sys/log/etcd-network/etcd-network.log

    Information similar to the following is displayed:
    2018-05-15 16:00:45.985584 C | etcdmain: database file (/var/etcd-data/etcd-network/etcd-network-2/member/snap/db index 16737187) does not match with snapshot (index 21909430).

    Or, information similar to the following is displayed:

    panic: xxx
    xxx/github.com/coreos/bbolt/xxx

    The command output shows that the etcd data is damaged.

  6. The path in the log is that on the etcd container. Run the following command to query the path on the node where the etcd container locates:

    cd /var/paas/run/etcd-network

    ls

    Information similar to the following is displayed:

    config.ini  etcd-network-2

  7. Run the following command to change the name of the data directory on the abnormal node:

    cd /var/paas/run/etcd-network

    mv etcd-network-2 etcd-network-2-old

    NOTE:

    In the preceding command, etcd-event-2 indicates the data directory on the abnormal etcd node in 6.

  8. Wait until the etcd instance is restarted by kubelet and run the following command to query the run log of the etcd instance:

    vi /var/paas/sys/log/etcd-network/etcd-network.log

    If the log of the critical level is not detected, the etcd data synchronization is complete.

  9. Query the etcd data directory and run the following command to confirm that the data directory is generated:

    cd /var/paas/run/etcd-network/etcd-network

    ls

    If the etcd instance is running properly, the fault is recovered. If it is still not restored, contact technical support for assistance.

Translation
Download
Updated: 2019-06-01

Document ID: EDOC1100062375

Views: 1403

Downloads: 12

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next