Removing an NVMe Drive

Procedure

• Before removing an NVMe drive:
  • Stop accessing the NVMe drive.
  • Back up date of the NVMe drive to prevent data loss.

• If multiple NVMe drives need to be replaced, remove them one by one. Do not remove multiple NVMe drives at the same time.
• Run command to uninstall the drive driver.
• Before removing a drive, you can use Smart Provisioning to delete data from the drive. For details, see "Erasing Hard Disks" in Smart Provisioning User Guide.

Windows OS (Huawei NVMe Drives)

For details about the Windows OSs supported by the server with NVMe drives, see "Search OSs" in the Computing Product Compatibility Checker.

1. Check whether a front bezel is installed.
   • If yes, go to 2.
   • If no, go to 3.

2. Remove the front bezel.

   For details, see Removing the Front Bezel.

3. Determine the slot and position of the drive.

   For details, see Drive Configurations and Drive Numbering.

4. Check the running status of the drive by observing its indicators.

   For details, see NVMe Drive Indicators.

5. Determine whether the drive needs to be removed.
   • If yes, go to 6.
   • If no, rectify the drive fault.

6. Orderly hot swap the drive. For details, see "Hot Swap > Orderly Hot Swap > Orderly Hot Swap of a Drive in Windows" in ES3000 V5 NVMe PCIe SSD User Guide.
7. Observe the indicator status of the NVMe drive. If the green indicator is off and the yellow indicator blinks at 0.5 Hz, remove the NVMe drive slowly.

   For details about the drive indicator status, see NVMe Drive Indicators.

   For details about how to manually remove an NVMe drive, see Removing a SAS/SATA Drive.

8. Place the removed component in an ESD bag.

   Place only one component in an ESD bag.

9. Install a drive filler.

   Perform this operation only when a drive is not installed immediately.

Windows OS (Non-Huawei NVMe Drives)

For details about the Windows OSs supported by the server with NVMe drives, see "Search OSs" in the Computing Product Compatibility Checker.

1. Check whether a front bezel is installed.
   • If yes, go to 2.
   • If no, go to 3.

2. Remove the front bezel.

   For details, see Removing the Front Bezel.

3. Determine the slot and position of the drive.

   For details, see Drive Configurations and Drive Numbering.

4. Check the running status of the drive by observing its indicators.

   For details, see NVMe Drive Indicators.

5. Determine whether the drive needs to be removed.
   • If yes, go to 6.
   • If no, rectify the drive fault.

6. Open Computer Management on the local PC.

   The Computer Management window is displayed.

7. In the navigation pane on the left, choose Device Manager.

   The Device Manager window is displayed.

8. Double-click Storage controllers to open the storage device list.
9. Select the NVMe drive to be removed.
10. Right-click and choose Uninstall device.

    The uninstalled NVMe drive is not displayed in Storage controllers after the removal.

11. In the navigation pane on the left, choose Storage > Disk Management.

    Ensure that no removed NVMe drive is displayed in the Disk Management window.

12. Observe the indicator status of the NVMe drive. If the green indicator is off and the yellow indicator blinks at 0.5 Hz, remove the NVMe drive slowly.

    For details about the drive indicator status, see NVMe Drive Indicators.

    For details about how to manually remove an NVMe drive, see Removing a SAS/SATA Drive.

13. Place the removed component in an ESD bag.

    Place only one component in an ESD bag.

14. Install a drive filler.

    Perform this operation only when a drive is not installed immediately.

Linux OS

• For details about the Linux OSs supported by the server with NVMe drives, see "Search OSs" in the Computing Product Compatibility Checker
• The kernel of the RHEL 7.3 OS must be upgraded to kernel-3.10.0-514.26.2.e17.x86_64 or later to support orderly hot swap of NVMe drives.
• The following uses RHEL 7.3 as an example. The operations apply to RHEL 7.3 and RHEL 7.4.

1. Check whether a front bezel is installed.
   • If yes, go to 2.
   • If no, go to 3.

2. Remove the front bezel.

   For details, see Removing the Front Bezel.

3. Determine the slot and position of the drive.

   For details, see Drive Configurations and Drive Numbering.

4. Check the running status of the drive by observing its indicators.

   For details, see NVMe Drive Indicators.

5. Determine whether the drive needs to be removed.
   • If yes, go to 6.
   • If no, rectify the drive fault.

6. If the operating system is RHEL 7.3, upgrade the kernel to kernel-3.10.0-514.26.2.e17.x86_64 or later.
   1. Log in to the server whose OS kernel is to be upgraded as user root.
   2. Check the current kernel version.

      uname -a

   3. Download the kernel upgrade package to the local PC.

      Log in to the official Red Hat website using a customer portal account and download the kernel upgrade package (kernel-3.10.0-514.26.2.el7.x86_64.rpm for example).

      Link: https://access.redhat.com/downloads/content/rhel---7/x86_64/2456/kernel/3.10.0-514.26.2.el7/x86_64/fd431d51/package

   4. Use the KVM to mount the directory where the upgrade package is located.
      Figure 8-23 Mounting the directory
      
   5. Mount the upgrade package to a directory on the OS, for example, /home.

      mount /dev/sr0 /home

   6. Check the upgrade package in the /home directory.

      cd /home

      ls kernel*

   7. Upgrade the system kernel.

      rpm -ivh kernel*.x86_64.rpm

   8. Run the reboot command to restart the server and go to the new kernel.

      reboot

   9. Verify the upgrade.

      uname -r

      The following information is displayed:

      kernel-3.10.0-514.26.2.el7.x86_64

7. Configure kernel parameters.
   1. Log in to the OS, and open /boot/efi/EFI/redhat/grub.cfg (if the OS is installed in UEFI mode) or /boot/grub2/grub.cfg (if the OS is installed in common mode).

      For example, in UEFI mode, run the vi /boot/efi/EFI/redhat/grub.cfg command to open the file.

      Figure 8-24 Opening the configuration file
      
   2. Press Insert.
   3. Locate "linuxefi /vmlinuz-3.10.0-514.26.2.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8" and add "pciehp.pciehp_force=1 pci=pcie_bus_perf" after it.

      If the kernel is upgraded, add the content to the end of the kernel parameter after the upgrade. Ensure that there is a space between the added content and the preceding content, but the content cannot be in a new line.

      Figure 8-25 Configuring kernel parameters
      
   4. Press Esc.
   5. Enter :wq.

      Save the modification and exit.

   6. Restart the server.

8. If the operating system is RHEL 7.3 or RHEL 7.4, change the value of the a8 register. Otherwise, the NVMe PCIe drives do not support orderly hot swap.

   Skip this step if the server has 24 x 2.5" NVMe drives. You do not need to change the a8 register value for the server with 24 x 2.5" NVMe drives.

   1. Run the following command to query the device (B/D/F) of the NVMe drive:
      dmidecode -t 9

      Figure 8-26 Device (B/D/F) of drive 2 (example)
      

      The device (B/D/F) of drive 2 is 0000:21:02.0.

   2. Run the following command to query the root port (B/D/F) of the NVMe drive:

      lspci -tv|grep -i device -5

      device: indicates the device (B/D/F) of the NVMe drive.

      For example, to query the root port (B/D/F) of drive 2, run the following command:

      lspci -tv|grep -i 21 -5

      Figure 8-27 Root port (B/D/F)
      

      The root port (B/D/F) of drive 2 is 21:03.0.

   3. Modify the a8 register value.
      1. Run the following command to record the original a8 register value (the PCI data in column 9 of row a0): see Figure 8-28.

         lspci -s <B/D/F> -xxx

         <B/D/F>: indicates the root port (B/D/F) of the NVMe drive. For details about how to obtain it, see 8.b.

         For example, if the drive No. is 2, run the following command:

         lspci -s 21:03.0 -xxx

         Figure 8-28 Querying the original register value
         

         If the default value of the a8 register is not f1, contact technical support.

      2. Change the value of the a8 register to e1.

         setpci -s <B/D/F> a8.B=e1

         <B/D/F>: indicates the root port (B/D/F) of the NVMe drive. For details about how to obtain it, see 8.b.

      3. Run the following command to check the modification result:

         lspci -s <B/D/F> -xxx

         <B/D/F>: indicates the root port (B/D/F) of the NVMe drive. For details about how to obtain it, see 8.b.

         Figure 8-29 Querying the new register value of drive 2 (example)
         

9. Stop accessing the NVMe drive.

   If a file system is mounted to the NVMe drive, run the following command to unmount the file system from it:

   umount /dev/NVMe drive letter

10. Obtain the mapping between the server drive IDs and the slot IDs on the operating system.

    For details about the drive numbers and positions, see Drive Numbering.

    Table 8-3 Mapping

    Configuration	Drive No.	Slot ID
    24 x 2.5" NVMe drive configuration	0 to 23	80 to 103
    24 x 2.5" (16 x SAS/SATA + 8 x NVMe) drive configuration	0 to 3	80 to 83
    	20 to 23	84 to 87

11. Run the following command to safely remove the drive:

    echo n > /sys/bus/pci/slots/Slot ID/power

    • In the command, the value of n can be 0 or 1. The value 0 indicates unloading a drive, and the value 1 indicates loading a drive. You can load a drive only before the unloaded drive is removed.
    • Slot ID: see Table 8-3.

    For example, to orderly hot remove the NVMe drive in slot 3, run the following command:

    echo 0 > /sys/bus/pci/slots/83/power

12. Observe the indicator status of the NVMe drive. If the green indicator is off and the yellow indicator blinks at 0.5 Hz, remove the NVMe drive slowly.

    For details about the drive indicator status, see NVMe Drive Indicators.

    For details about how to manually remove an NVMe drive, see Removing a SAS/SATA Drive.

13. Place the removed component in an ESD bag.

    Place only one component in an ESD bag.

14. If the operating system is RHEL 7.3 or SLES 12.2, run the following command to restore the original value of the a8 register:

    setpci -s <B/D/F> a8.B=Original a8 register value

    <B/D/F>: indicates the root port (B/D/F) of the NVMe drive. For details about how to obtain it, see 8.b.

    • Skip this step if the server has 24 x 2.5" NVMe drives. You do not need to change the a8 register value for the server with 24 x 2.5" NVMe drives.
    • If the value of the register is not restored to the original one, the orderly hot insertion of the NVMe drives may be abnormal.

15. Install a drive filler.

    Perform this operation only when a drive is not installed immediately.