No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

HUAWEI CLOUD Stack 6.5.0 Troubleshooting Guide 02

Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Pod in the CrashLoopBackOff State

Pod in the CrashLoopBackOff State

Symptom

  • Pod is suspended and is in the CrashLoopBackOff state.
  • Pod functions are abnormal.

Possible Causes

The docker umount devicemapper exception occurs, causing devices mounted in Docker fail to be uninstalled and pod suspension.

Troubleshooting Method

  1. Use PuTTY to log in to the manage_lb1_ip node.

    The default username is paas, and the default password is QAZ2wsx@123!.

  2. Run the following command and enter the password of the root user to switch to the root user:

    su - root

    Default password: QAZ2wsx@123!

  3. Run the following command to check whether the pod is normal:

    kubectl get pods --all-namespaces|grep CrashLoopBackOff

    fst-manage        alm-website-1825011304-c7v32                     0/1       CrashLoopBackOff   205        11h

    The preceding command output shows that alm-website-1825011304-c7v32 is abnormal.

  4. Run the following command to query the IP address of the node where the faulty component resides:

    kubectl get pods PODNAME -o yaml -n NAMESPACE|grep hostIP

    PODNAME indicates the name of the abnormal pod. NAMESPACE indicates the namespace of the abnormal pod.

    HostIP: 10.109.220.168

    The preceding output shows that the IP address is 10.109.220.168.

  5. Check whether the abnormality is caused by the docker umount devicemapper exception.

    Log in to the node where the abnormal pod locates as user paas, switch to user root, and run the following command:

    journalctl -n

    Check whether there is the umount docker-thinpool exception.

    -- Logs begin at Wed 2019-03-06 06:00:01 CST, end at Thu 2019-03-07 14:44:05 CST. --
    Mar 07 14:43:55 ab-e02ff-2110780-31a75459.novalocal docker[13114]: time="2019-03-07T14:43:55.946662390+08:00" level=error msg="Attempting next endpoint fo
    Mar 07 14:43:55 ab-e02ff-2110780-31a75459.novalocal docker[13114]: time="2019-03-07T14:43:55.946695768+08:00" level=error msg="Handler for POST /v1.23/ima
    Mar 07 14:43:56 ab-e02ff-2110780-31a75459.novalocal sudo[18366]:     paas : TTY=unknown ; PWD=/etc/sv/monit ; USER=root ; COMMAND=/opt/kube-agent/kubernet
    Mar 07 14:43:56 ab-e02ff-2110780-31a75459.novalocal [18386]: [umount /dev/mapper/docker-8:2-787953-5c87ad402c4f391edd85d5849142b4264779dfdacce9c389dbee0fb
    Mar 07 14:43:59 ab-e02ff-2110780-31a75459.novalocal [18544]: [umount /dev/mapper/docker-8:2-787953-5c87ad402c4f391edd85d5849142b4264779dfdacce9c389dbee0fb
    Mar 07 14:44:02 ab-e02ff-2110780-31a75459.novalocal kernel: IPv4: martian source 192.255.179.202 from 192.255.179.1, on dev gw_7505d64a
    Mar 07 14:44:02 ab-e02ff-2110780-31a75459.novalocal kernel: ll header: 00000000: ff ff ff ff ff ff 70 79 90 7d ad 8f 08 06        ......py.}....
    Mar 07 14:44:03 ab-e02ff-2110780-31a75459.novalocal docker[13114]: time="2019-03-07T14:44:03.480499870+08:00" level=info msg="Calling POST /v1.23/containe
    Mar 07 14:44:03 ab-e02ff-2110780-31a75459.novalocal docker[13114]: time="2019-03-07T14:44:03.496676263+08:00" level=info msg="Calling POST /v1.23/containe
    Mar 07 14:44:05 ab-e02ff-2110780-31a75459.novalocal sudo[18843]:     paas : TTY=unknown ; PWD=/etc/sv/monit ; USER=root ; COMMAND=/opt/kube-agent/kubernet
    • If there is no umount docker-thinpool exception, the CrashLoopBackOff state is not caused by the docker umount devicemapper exception.
    • If the exception occurs, record the file system ID of devicemapper that fails to be deleted and perform the subsequent steps. For example, 5c87ad402c4f391edd85d5849142b4264779dfdacce9c389dbee0fb.

  6. Run the following command to query the information about the device failed to be deleted:

    mount|grep file system ID

    For example, mount|grep 5c87ad402c4f391edd85d5849142b4264779dfdacce9c389dbee0fb.

    file system ID indicates the file system ID you obtained in 5.

    Record the path of the device failed to be deleted.

    /dev/mapper/docker-253:2-1048580-5c87ad402c4f391edd85d5849142b4264779dfdacce9c389dbee0fb on /opt/docker/devicemapper/mnt/5c87ad402c4f391edd85d5849142b4264779dfdacce9c389dbee0fb type ext4 (rw,relatime,seclabel,stripe=128,data=ordered)

  7. Run the following command to uninstall the abnormal device:

    umount device path

    device path indicates the path you recorded in 6.

  8. Run the following command to obtain the ID of the container in dead stat:

    docker ps -a|grep dead

  9. Run the following command to delete the container:

    docker rm -f container id

    container id indicates the container ID you obtained in 8.

  10. Run the command in 3 to check whether the pod is recovered.

    If no command output is displayed, the pod is normal.

    If the pod is abnormal, contact technical support for assistance.

Translation
Download
Updated: 2019-06-01

Document ID: EDOC1100062375

Views: 2129

Downloads: 12

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next