No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionCloud 6.3.1.1 Troubleshooting Guide 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Failure in etcd Backup and Restoration Due to the Node Where the etcd-backup Service Is Deployed Is Powered Off or Restarted (Tenant Management Zone)

Failure in etcd Backup and Restoration Due to the Node Where the etcd-backup Service Is Deployed Is Powered Off or Restarted (Tenant Management Zone)

Symptom

The etcd backup in the management zone is successful. When one node of the etcd cluster is powered off, etcd fails to be restored, and the pod of etcd is working improperly. After the node is powered on again and etcd is rectified manually, etcd still fails to be restored.

Possible Causes

The node powered off is the node where the etcd-backup service is deployed. During etcd restoration, the etcd-backup service is scheduled to another node. As a result, the restoration fails.

Troubleshooting Method

  1. Use PuTTY to log in to the om_core1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  1. Run the following command to query the StatefulSets of the etcd services and check the number of the started etcd services:

    kubectl get statefulset -nmanage | grep etcd

    etcd                    3             3                        2d
    etcd-event              3             3                        2d
    etcd-network            3             0                        2d

    In the command output, the second column indicates the expected number of started pods, and the third column indicates the acutal number of started pods. Check whether the numbers are the same.

    • If yes, the servics are running properly.
    • If no, go to 3.

  2. Change the value of enable for StatefulSet of etcd-network to true. (The following picture uses the etcd-network pod failure as an example.)

    kubectl edit statefulset -nmanage etcd-network

  3. Run the following command to check whether the pod status is Running:

    kubectl get pod -nmanage -owide | grep etcd

    Go to the next step after the status of all pods is Running.

  4. Run the following command to obtain the node where the pod runs:

    kubectl get pod -nom -owide | grep etcd

  5. Log in to the node obtained in 5 and run the following command to switch to the root user:

    su - root

  6. Log in to the node obtained in 5 and run the following command to query the container ID of etcd:

    docker ps | grep etcd

  7. Obtain the value of Value mapping Key whose value is /registry/configmaps/manage/etcd.download.info after accessing the etcd container.

    docker exec -ti containerID bash

    ETCDCTL_API=3 /start-etcd --cacert /srv/kubernetes/ca.crt --cert

    /srv/kubernetes/server.cer --key

    /srv/kubernetes/server_key.pem --endpoints

    https://127.0.0.1:4001 get /registry/configmaps/manage/etcd.download.info

    containerID is the container ID obtained in 7.

    {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"etcd.download.info","namespace":"manage","selfLink":"/api/v1/namespaces/manage/configmaps/etcd.download.info","uid":"15b788c6-d9a1-11e8-8f76-286ed488c964","creationTimestamp":"2018-10-27T04:30:51Z","labels":{"app":"etcd-backup-1569e6e2-d9a","appgroup":"etcd-manage_default-appGroup","stack-name":"etcd-manage"},"enable":true},"data":{"etcd":"{\"etcdtype\":\"etcd\",\"downloadurl\":\"https://etcd-backup.manage.svc.cluster.local:30436/pbs/v2/manage/etcd/tar/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"bakdatainfo\":\"/ftpboot/etcd_backup/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"backupway\":\"sftp\",\"encryptionflag\":true,\"remoteaddress\":{\"Id\":1,\"ip\":\"9.91.0.80\",\"name\":\"9.91.0.45\",\"password\":\"QAZ2wsx@123!\",\"path\":\"/ftpboot\",\"port\":22,\"type\":\"sftp\",\"user\":\"sftpuser\"}}","etcd-network":"{\"etcdtype\":\"etcd-network\",\"downloadurl\":\"https://etcd-backup.manage.svc.cluster.local:30436/pbs/v2/manage/etcd-network/tar/etcd-network_2018-10-29-11-20-21_manage_825eaa88f2a53550895dfb962c653436.tar.gz\",\"bakdatainfo\":\"/ftpboot/etcd_backup/etcd-network_2018-10-29-11-20-21_manage_825eaa88f2a53550895dfb962c653436.tar.gz\",\"backupway\":\"sftp\",\"encryptionflag\":true,\"remoteaddress\":{\"Id\":1,\"ip\":\"9.91.0.80\",\"name\":\"9.91.0.45\",\"password\":\"QAZ2wsx@123!\",\"path\":\"/ftpboot\",\"port\":22,\"type\":\"sftp\",\"user\":\"sftpuser\"}}"}}

  8. Delete the data in the data field that maps the Key field if etcd, etcd-event, or etcd-network fails to be restored.

    For example, if etcd-network fail to be restored, delete the data of etcd-network in the data field.

    {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"etcd.download.info","namespace":"manage","selfLink":"/api/v1/namespaces/manage/configmaps/etcd.download.info","uid":"15b788c6-d9a1-11e8-8f76-286ed488c964","creationTimestamp":"2018-10-27T04:30:51Z","labels":{"app":"etcd-backup-1569e6e2-d9a","appgroup":"etcd-manage_default-appGroup","stack-name":"etcd-manage"},"enable":true},"data":{"etcd":"{\"etcdtype\":\"etcd\",\"downloadurl\":\"https://etcd-backup.manage.svc.cluster.local:30436/pbs/v2/manage/etcd/tar/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"bakdatainfo\":\"/ftpboot/etcd_backup/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"backupway\":\"sftp\",\"encryptionflag\":true,\"remoteaddress\":{\"Id\":1,\"ip\":\"9.91.0.80\",\"name\":\"9.91.0.45\",\"password\":\"QAZ2wsx@123!\",\"path\":\"/ftpboot\",\"port\":22,\"type\":\"sftp\",\"user\":\"sftpuser\"}}","etcd-network":"{\"etcdtype\":\"etcd-network\",\"downloadurl\":\"https://etcd-backup.manage.svc.cluster.local:30436/pbs/v2/manage/etcd-network/tar/etcd-network_2018-10-29-11-20-21_manage_825eaa88f2a53550895dfb962c653436.tar.gz\",\"bakdatainfo\":\"/ftpboot/etcd_backup/etcd-network_2018-10-29-11-20-21_manage_825eaa88f2a53550895dfb962c653436.tar.gz\",\"backupway\":\"sftp\",\"encryptionflag\":true,\"remoteaddress\":{\"Id\":1,\"ip\":\"9.91.0.80\",\"name\":\"9.91.0.45\",\"password\":\"QAZ2wsx@123!\",\"path\":\"/ftpboot\",\"port\":22,\"type\":\"sftp\",\"user\":\"sftpuser\"}}"}}

  9. Re-write the deleted data.

    ETCDCTL_API=3 /start-etcd --cacert /srv/kubernetes/ca.crt --cert /srv/kubernetes/server.cer --key /srv/kubernetes/server_key.pem --endpoints https://127.0.0.1:4001 put /registry/configmaps/manage/etcd.download.info '{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"etcd.download.info","namespace":"manage","selfLink":"/api/v1/namespaces/manage/configmaps/etcd.download.info","uid":"15b788c6-d9a1-11e8-8f76-286ed488c964","creationTimestamp":"2018-10-27T04:30:51Z","labels":{"app":"etcd-backup-1569e6e2-d9a","appgroup":"etcd-manage_default-appGroup","stack-name":"etcd-manage"},"enable":true},"data":{"etcd":"{\"etcdtype\":\"etcd\",\"downloadurl\":\"https://etcd-backup.manage.svc.cluster.local:30436/pbs/v2/manage/etcd/tar/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"bakdatainfo\":\"/ftpboot/etcd_backup/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"backupway\":\"sftp\",\"encryptionflag\":true,\"remoteaddress\":{\"Id\":1,\"ip\":\"9.91.0.80\",\"name\":\"9.91.0.45\",\"password\":\"QAZ2wsx@123!\",\"path\":\"/ftpboot\",\"port\":22,\"type\":\"sftp\",\"user\":\"sftpuser\"}}"}}'
    NOTE:

    In the preceding command, {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"etcd.download.info","namespace":"manage","selfLink":"/api/v1/namespaces/manage/configmaps/etcd.download.info","uid":"15b788c6-d9a1-11e8-8f76-286ed488c964","creationTimestamp":"2018-10-27T04:30:51Z","labels":{"app":"etcd-backup-1569e6e2-d9a","appgroup":"etcd-manage_default-appGroup","stack-name":"etcd-manage"},"enable":true},"data":{"etcd":"{\"etcdtype\":\"etcd\",\"downloadurl\":\"https://etcd-backup.manage.svc.cluster.local:30436/pbs/v2/manage/etcd/tar/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"bakdatainfo\":\"/ftpboot/etcd_backup/etcd_2018-10-29-11-20-20_manage_e025db26f9bd8c747a6db15f8f796386.tar.gz\",\"backupway\":\"sftp\",\"encryptionflag\":true,\"remoteaddress\":{\"Id\":1,\"ip\":\"9.91.0.80\",\"name\":\"9.91.0.45\",\"password\":\"QAZ2wsx@123!\",\"path\":\"/ftpboot\",\"port\":22,\"type\":\"sftp\",\"user\":\"sftpuser\"}}"}} indicates the Key value after the cluster that fails to be restored is deleted from the data field.

    Information similar to the following is displayed:

    OK

  10. Run the following command to exit the container:

    exit

  11. Perform the restoration operation in the OM zone.

    If the restoration is successful, the fault is rectified.

Translation
Download
Updated: 2019-06-10

Document ID: EDOC1100063248

Views: 22700

Downloads: 37

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next