No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

HUAWEI CLOUD Stack 6.5.0 Troubleshooting Guide 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Insufficient Docker Thinpool Space

Insufficient Docker Thinpool Space

Symptom

The following error information is displayed during the health check of the docker thinpool space.

10.109.67.113:
Process exited with status 10.109.67.113:
Process exited with status 1

Possible Causes

The docker thinpool space is insufficient.

Troubleshooting Method

  1. Log in to the abnormal node as the paas user. Run the following command to switch to the root user:

    su - root

  2. Check whether any new disk exists.

    Run the fdisk -l command to check whether any new disk exists (If no new disk exists, attach an independent one):

    Assume that the newly added disk is /dev/sdh1.
    • If a new scale-out disk is added, go to 3.
    • If no new disk is added, go to 10.

  3. Disable the docker service:

    systemctl stop docker

  4. Check the name of the new scale-out disk.

    fdisk -l

  5. Run the following command to query the docker VG name:

    vgdisplay

      --- Volume group ---
      VG Name               opt_vg
      System ID
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  2
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                1
      Open LV               1
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               100.00 GiB
      PE Size               4.00 MiB
      Total PE              25599
      Alloc PE / Size       25575 / 99.90 GiB
      Free  PE / Size       24 / 96.00 MiB
      VG UUID               P9SWQ3-5Jmr-QFo7-4R38-llLC-J85n-gMYxuD
      --- Volume group ---
      VG Name               vg-docker
      System ID
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  6
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                1
      Open LV               1
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               100.00 GiB
      PE Size               4.00 MiB
      Total PE              25599
      Alloc PE / Size       24757 / 96.71 GiB
      Free  PE / Size       842 / 3.29 GiB
      VG UUID               vT44E3-INMa-Kl26-dT5t-Z0cW-aEy4-miKd61

    Record the VG name with the keyword docker. Here, vg-docker is used as an example.

  6. Scale out the VG.

    vgextend vg-docker /dev/sdh1

    In the preceding command,vg-docker indicates the VG name queried in 5.

    /dev/sdh1 indicates the new disk name queried in 4. The new disk name can be changed as required.

  7. Scale out the LV.

    lvextend -l+100%FREE -n vg-docker/data

    In the preceding command,vg-docker indicates the queried VG name.

    The free space of 100% is directly allocated to docker/data, which can be adjusted.

    Run the following command to specify the scale-out size:

    lvextend -L+16M -n vg-docker/data

    The extended value indicates docker size, as shown in the following command output.

    The above figure indicates that "Logical volume data not found in volume group docker" can be ignored, and has no effect on the final operation result.

  8. Restart the dm thin pool.

    1. Obtain the name of the docker thin pool.

      dmsetup status | grep pool

      docker-253:1-219956224-pool: 0 31457280 thin-pool 48 435/524288 17593/245760 - rw discard_passdown queue_if_no_space

      In this command output, docker-253:1-219956224-pool indicates the name of the docker thin pool service.

    2. Obtain the thin pool table.

      dmsetup table docker-253:1-219956224-pool

      0 31457280 thin-pool 253:3 253:2 128 32768 1 skip_block_zeroing

      In this command output, docker-253:17-1835016-pool indicates the name of the docker thin pool obtained in 8.a.

      31457280 indicates the number of sectors.

      thin-pool 253:3 253:2 128 32768 1 skip_block_zeroing indicates the thin pool table.

    3. Calculate the size of the newly added disk.

      Only the size of the thin pool table (the second field, that is, the 31457280 field in the preceding command output) can be changed. This field indicates the number of sectors.

      Run the following command to check the size of one sector:

      lvs

      LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
        thinpool docker twi-aot--- 94.90g             22.35  2.33                     
        vol_opt  opt_vg -wi-ao---- 99.90g       

      In this command output, thinpool indicates the LV name, and docker indicates the VG name.

      blockdev --getpbsz/dev/VG name/LV name

      512

      The increased number of sectors can be obtained when the size (bytes) of the added disk is divided by 512. Replace the original sector number with the new number (the sum of the original sector number and the increased sector number), and then update the thin pool table.

    4. Restart the dm thin pool service.

      dmsetup suspend docker-253:17-1835016-pool

      In the preceding command, docker-253:17-1835016-pool indicates the name of the docker thin pool service obtained in 8.a.

      dmsetup reload docker-253:17-1835016-pool --table '0 62914560 thin-pool 253:3 253:2 128 32768 1 skip_block_zeroing

      In the preceding command, thin-pool 253:3 253:2 128 32768 1 skip_block_zeroing indicates the thin pool table obtained in 8.b.

      62914560 indicates the modified sector number.

      dmsetup resume docker-253:17-1835016-pool

      If this command is executed, only the number of sector is changed. In this way, the size of the thin pool is scaled out. Then restart the docker.

    5. Check whether the scaling out of the VG size of the docker thinpool is successful.

      vgdisplay

      The added value of VG Size indicates the disk scale-out size (if the value of VG Size is added, the disk size scaling out succeeds), as shown in the following command output:

       --- Volume group ---
        VG Name               vg-docker
        System ID
        Format                lvm2
        Metadata Areas        2
        Metadata Sequence No  7
        VG Access             read/write
        VG Status             resizable
        MAX LV                0
        Cur LV                1
        Open LV               1
        Max PV                0
        Cur PV                2
        Act PV                2
        VG Size               124.99 GiB
        PE Size               4.00 MiB
        Total PE              31998
        Alloc PE / Size       24757 / 96.71 GiB
        Free  PE / Size       7241 / 28.29 GiB
        VG UUID               EU72rf-mP1G-VO99-7X5h-DsfI-VfzG-O1Z8q3

  9. Restart the docker.

    systemctl restart docker

    No further action is required.

  10. Scale out dm.

    If no new disk is attached to the server but Logical Volume Manager (LVM) has enough space, skip 6 and perform 3, 7, 8, and 9 to scale out the LV to achieve scale-out dm.

Translation
Download
Updated: 2019-06-01

Document ID: EDOC1100062375

Views: 1229

Downloads: 12

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next