How Do I Replace the NTP Server in the FusionInsight Cluster

Question

When the NTP server of the cluster is changed or modified, how do I specify a new NTP server in the cluster to ensure that the cluster time can be synchronized with the new NTP clock source?

Answer

• Replacing the NTP server is a critical operation and may result in time change in the cluster.
• Before replacing the NTP server, if the time difference between the NTP server and the cluster is greater than 150 seconds, you need to restart the cluster to prevent data loss. Service interruption occurs when the cluster is stopped.

1. Log in to FusionInsight Manager and check whether any alarm is not cleared.
   • If there are uncleared alarms, see FusionInsight HD Troubleshooting to handle the alarms. After the alarms are handled, go to 2.
   • If there is no uncleared alarm, go to 2.

2. Use PuTTY to log in to the active and standby management nodes as the root user.

   After FusionInsight and controller are co-deployed or the FusionInsight server is security-hardened, log in to the FusionInsight server as the sopuser user and run the su - root command to switch to the root user.

   The default username and password are available in iMaster NCE-Campus Default Usernames and Passwords (Enterprise Network or Carrier). If you have not obtained the access permission of the document, see Help on the website to find out how to obtain it.

3. (Optional) Configure NTP security authentication.

   If the authentication function is enabled on the NTP server, you need to configure NTP security authentication and obtain the authentication key from the NTP server administrator.

   The operation does not need to be performed on the standby management node.

   1. Run the following command to go to the related directory:

      cd ${BIGDATA_HOME}/om-server/om/bin/tools/

   2. Run the following command to configure the NTP authentication key used for accessing the third-party NTP server from a cluster.

      sh authenticateNtp.sh --ntp_server_ip ip_address

      ip_address: indicates the IP address of the third-party NTP server. It does not support multiple IP addresses. That is, if there are multiple NTP servers, you need to run the command for each NTP server for authentication.

      For example, run the following command to access the third-party NTP server with the IP address 192.168.1.1. The authentication key is 10 M b273290137CH.

      sh authenticateNtp.sh --ntp_server_ip 192.168.1.1

      Enter the NTP authentication key as prompted, and press Enter. The authentication key index value cannot be 1. You are advised to use other values.

      Please input 192.168.1.1 authentication_code: 10 M b273290137CH

      If the following information is displayed, the NTP security authentication is configured successfully.

      Success to Authenticate ntp server to 192.168.1.1

4. Run the following command on the primary management node to check the management plane gateway:

   cat ${OMS_RUN_PATH}/workspace/conf/oms-config.ini | grep ws_gateway

5. On the primary and secondary management nodes, check whether the nodes are connected to the management plane gateway.
   • If so, go to 6.
   • If not, contact the network administrator to rectify the network fault. After exceptions are handled, go to 6.

6. On the primary management node, check the time difference between the new NTP server and the cluster. The unit is second.
   For example, to check the time difference with the NTP server at 10.1.1.1, run the su - root command and enter the password to switch to the root user, and run the ntpdate -d 10.1.1.1 command. The following information is displayed.

   31 Jan 10:10:10 ntpdate[6015]: adjust time server 10.1.1.1 offset -0.000020 sec

   In the information, -0.000020 indicates the time difference. If the value is a positive number, the time of the NTP server is earlier than the time of the cluster. If the value is a negative number, the time of the NTP server is later than the time of the cluster.

7. Check whether the absolute value of the time difference exceeds 150.
   • If so, go to 8.
   • If not, perform 10 as the omm user.

8. Stop the controller and FusionInsight cluster.
   1. Stop the controller cluster.
      1. Log in to the iMaster NCE-Campus management plane as the admin user.
      2. On the home page, click iMaster_NCE-Campus.
      3. Choose Stop > Stop All.

   2. Stop the FusionInsight cluster.
      1. Log in to the FusionInsight management portal as the admin user.
      2. Stop the cluster on the home page.

9. Check whether the time of the NTP server time is earlier than the time of the cluster.
   • If so, after the cluster portal displays Operation succeeded, wait until the time difference obtained in 6 is due and perform 11 as the omm user.
   • If not, after the cluster portal displays Operation succeeded, perform 11 as the omm user directly.

10. Run the following command on the primary management node to replace the NTP server. No further action is required.

    sh ${BIGDATA_HOME}/om-server/om/bin/tools/modifyntp.sh --ntp_server_ip 10.1.1.1

    A maximum of two NTP servers in active/standby mode can be configured. Use a comma (,) to separate the servers, for example, sh ${BIGDATA_HOME}/om-server/om/bin/tools/modifyntp.sh --ntp_server_ip 10.1.1.1,10.1.1.2.

    Run the ntpdate -d 10.1.1.1 command to check whether the network between the FusionInsight server and the NTP server is normal.

    If the network is disconnected, run the service network stop and service network start commands to restart the network service.

11. Run the following command on the primary management node to forcibly synchronize time from the NTP server 10.1.1.1 immediately and replace the NTP server.

    sh ${BIGDATA_HOME}/om-server/om/bin/tools/modifyntp.sh --ntp_server_ip 10.1.1.1 --force_sync_time

    A maximum of two NTP servers in active/standby mode can be configured. Use a comma (,) to separate the servers, for example, sh ${BIGDATA_HOME}/om-server/om/bin/tools/modifyntp.sh --ntp_server_ip 10.1.1.1,10.1.1.2 --force_sync_time.

12. Synchronize the controller server time with the FusionInsight time.

    If FusionInsight and iMaster_NCE-Campus are co-deployed, perform this step only on the node where FusionInsight is not installed.

    1. Log in to the controller server as the sopuser user and switch to the root user.
    2. Run the following command to synchronize the time.

       /bin/systemctl stop ntpd.service

       ntpdate IP address of the primary FusionInsight management node

       /bin/systemctl start ntpd.service

    3. Perform 12.a to 12.b on each controller server to synchronize the FusionInsight time.

13. If the cluster is stopped, start the cluster after the NTP server is replaced.
    1. Start the FusionInsight cluster.
       1. Log in to the FusionInsight management portal as the admin user.
       2. Start the cluster on the home page.

    2. Start the controller cluster.
       1. Log in to the iMaster NCE-Campus management plane as the admin user.
       2. On the home page, click iMaster_NCE-Campus.
       3. Choose Start > Start All.