Geographic Redundancy

Common concepts

Differences between the active and standby sites

Manual Switchover

Solution introduction:

The primary and secondary sites communicate with each other through heartbeat links and detect the status of the peer site in real time. The primary site synchronizes product data to the secondary site in real time through the data replication link to ensure product data consistency between the primary and secondary sites.

When a disaster occurs at the primary site, perform the takeover operation at the secondary site. The secondary site becomes the active site and provides services externally. The primary site becomes standby.

Manual switchover trigger conditions:

• The disaster such as an earthquake, fire, or power failure occurs at the primary site caused the system as a whole to be unable to provide services.
• The primary site is faulty, causing some key nodes to be damaged and unable to provide corresponding services. For example, database node (DB) corruption, platform service node (Common_Service) corruption, management domain service node (NMS) corruption, control domain service node (Controller or TController) corruption.

Solution schematic diagram:

Figure 3-23 Working principle of manual switchover in the DR system

The DR network can reuse the original network of iMaster NCE-Campus to reduce the network configuration of the primary and secondary sites.

Automatic Switchover (with Arbitration Service)

Solution introduction:

The arbitration service periodically checks the connectivity between the primary, secondary, and third-party site, and share the check results through data sharing links. When the network connection is abnormal or a site fault causes an arbitration heartbeat exception, the arbitration service selects the optimal site in the network based on the internal algorithms to perform an active/standby switchover.

Automatic switchover trigger conditions:

• A disaster such as an earthquake, fire, or power failure occurs at the primary site, and the fault is not rectified within the specified time.
• The heartbeat link between the primary and secondary sites is interrupted, and the data sharing link between the primary site and the third-party site is interrupted.
• In the manager+controller+analyzer compact deployment and manager deployment scenarios:
  • If any of the default key microservices of the system is faulty, the DR system triggers an automatic switchover to ensure normal service running.
  • If the service network (southbound or northbound network) is faulty due to a network port fault on the server, the system automatically triggers a switchover.
  • If all database instances are faulty, the system automatically triggers a switchover.

  • Manager+Controller+Analyzer deployment scenarios, nodes and application services are deployed in active/standby or cluster mode, and local protection is configured. Key microservice failover, server service network ports failover and all database instances failover are not separately configured.

  • The priorities of triggering an automatic switchover are as follows: All database instances are faulty > Server service network ports are faulty > Key microservices are faulty. If all database instances at the secondary site are faulty, an automatic switchover is not triggered even if key microservices at the primary site are faulty.

Arbitration service deployment:

• The CPU architecture of the primary site, secondary site and third-party site is required to be consistent. If the primary and secondary sites are ARM architecture servers, the third-party site is also required to be ARM architecture server.
• iMaster NCE-Campus in Manager+Controller+Analyzer deployment scenarios adopts five-node arbitration service deployment. The arbitration service is deployed at three sites in 2+2+1 mode.
  • Two arbitration nodes are deployed at both the primary site and secondary site. It is recommended that the two arbitration nodes be deployed on the Common_Service node. The arbitration nodes between the two sites are mutually protected. One arbitration node is deployed at the third-party site.
  • ETCD is deployed on the five arbitration nodes to form an ETCD cluster. Monitor is deployed on the four nodes of the primary site and secondary site, which monitors the network connectivity between sites and saves the results in the ETCD cluster.
  Figure 3-24 A five-node DR system
  
• iMaster NCE-Campus in manager+controller+analyzer compact deployment and manager deployment scenarios adopts three-node arbitration service deployment. The arbitration service is deployed at three sites in 1+1+1 mode.
  • One arbitration node is deployed at the primary site. One arbitration node is deployed at the secondary site. It is required that the arbitration node be deployed on the Common_Service node in manager+controller+analyzer compact deployment scenarios, and the arbitration node be deployed on the NMS_Server node in manager deployment scenarios. One arbitration node is deployed at the third-party site.
  • ETCD is deployed on the three arbitration nodes to form an ETCD cluster. Monitor is deployed on the two nodes of the primary site and secondary site, which monitors the network connectivity between sites and saves the results in the ETCD cluster.
  Figure 3-25 A three-node DR system
  

Solution schematic diagram:

The DR network can reuse the original network of iMaster NCE-Campus to reduce the network configuration of the primary and secondary sites.

Prerequisites

Before configuring the DR system, check that the configurations of the primary and secondary sites meet the following requirements.

• Deployment solution requirements:
  • Select the DR system as the protection site to ensure the DR services are the same.
  • The deployment schemes (such as product version, Product, component solution) of the two sites must be the same. When the deployment schemes of the two sites are the same, the software packages, number and specifications of the nodes at the two sites, management plane and product languages, versions, services, service versions, the root certificate of the DR system certificates, root keys, working keys, and the database user passwords of the product are the same by default.
  • Configure the NTP server for the primary and secondary sites to ensure the UTC time consistency. The time zone for the primary site can be different from that for the secondary site. Set the time zone according to the actual situation.
  • A CA certificate is dynamically generated when the management plane is installed. So you should update the CA certificate to ensure that the CA certificates at the two sites are consistent. For details, see Uploading and Updating CA Certificates (DR Scenario).
• Networking requirements:
  • The communication of the Inter-node communication IP address between the primary and secondary sites is normal.
  • The IP addresses of all nodes, excluding the NTP servers, at the primary and secondary sites are different. The IP address version must be the same. If IPv6 addresses are planned, the primary and secondary sites must use IPv6 addresses.
  • The bandwidth between the primary and secondary sites must meet the requirements. For details about bandwidth requirements, see "HLD > Bandwidth Planning" in EasySuite.
• iMaster NCE-Campus requirements: All services and database instances at the primary and secondary sites are normal. For details, see System Monitoring.

• The primary and secondary sites use different IP addresses to interconnect with an OSS. It is advisable to preconfigure the secondary site information on the OSS so that the secondary site can take over services when the primary site is down. If secondary site information cannot be preconfigured, change the currently connected IP address to the IP address of the secondary site after switchover in the DR system.
• In the Centralized deployment scenario, if the floating IP address for connecting to the northbound network is configured on the management node, ensure that the NIC for the floating IP address for connecting to the northbound network at the secondary site is disabled.
  1. Use PuTTY to log in to the management node at the secondary site as the ossadm user in SSH mode.
  2. Run the following command to query the NIC usage of the management node:

     > ifconfig

  3. If the query result contains the NIC for the floating IP address for connecting to the northbound network (for example, bond0:1), the NIC is not disabled. Run the following command to disable the NIC:

     > ifconfig bond0:1 down

  4. Run the following command again to query the NIC usage of the management node. The query result does not contain the NIC for the floating IP address for connecting to the northbound network.

     > ifconfig

• You have logged in to the management plane at the primary or secondary site. For details, see Logging In to the Management Plane.

Context

The precheck verifies that:

• The communication of the heartbeat links and data replication link between the primary and secondary sites is normal.

  If the primary and secondary sites can communicate with each other, the heartbeat links between the two sites must be normal and the DR system certificates are consistent between the two sites.

• The node quantity is consistent between the primary and secondary sites.
• Services and their versions are consistent between the primary and secondary sites.
• The CA certificate is consistent between the primary and secondary sites.
• The root key and working keys are consistent between the primary and secondary sites, respectively.
• The password for the same database instance user is consistent between the primary and secondary sites.
• All database instances and local replication status are normal at the primary and secondary sites.
• The language of the management plane is consistent between the primary and secondary sites.
• The time difference between the primary and secondary sites is less than 1 minute.
• The enabling status of the SSL protocol of the local master and slave database instances is consistent between the primary and secondary site products.

• Verify that the installation interval between the primary and secondary sites is less than 7 days.

  During initial DR creation, if the installation interval between the primary and secondary sites is greater than 7 days, the site installed later cannot be used as the primary site. If the site installed later is used as the primary site, the system displays the message "The newly installed cluster cannot be used as the primary cluster." This prevents data loss of the original cluster when the new cluster is used as the primary cluster in the site reconstruction scenario.

Precautions

• Perform operations provided in this section only at one site.
• If a backup or restoration task is in progress, perform operations in this section after the task is complete. Otherwise, the task or the DR operation may fail.

Procedure

1. On the management plane, choose HA > Remote High Availability System Management > Manage DR System from the main menu.
2. On the Manage Remote DR System page, click Configure DR System.
3. Refer to Table 3-119 to configure the site.
   Table 3-119 DR system parameters

   Parameter	Description
   Site name	Identifier of a site.
   Heartbeat IP address	IP address of the management node used for heartbeat communication, that is, IP address of the management nodes in the DR network. In the Distributed scenario, set the heartbeat IP addresses to the IP address in the DR network of OMP_01 and OMP_02 nodes at the primary and secondary sites respectively. In the centralized scenario, set the heartbeat IP addresses to the IP address in the DR network of NMS_Server node at the primary and secondary sites respectively. NOTE: The DR network can reuse the inter-node communication network or northbound network or use an independent network. The heartbeat IP address and replication IP address must be on the same network plane. Obtain the heartbeat IP address based on the site requirements.

4. Click Add Product. Select the primary and secondary site products, and the data replication direction. Click Precheck.
   • If yes, the configuration requirements for the DR system are met, click Save Draft.
   • If no, resolve the problems as prompted.

Prerequisites

You have logged in to the management plane at the primary and secondary sites. For details, see Logging In to the Management Plane.

Precautions

• After the primary and secondary sites are deleted, the DR relationship between products at the two sites are deleted. In this case, the secondary site no longer provides DR protection for the primary site, and data is no longer synchronized between the two sites, but the data at the two sites is not deleted.
• After the primary and secondary sites are deleted, the historical health check records of the sites are deleted.
• If a backup or restoration task is in progress, perform operations in this section after the task is complete. Otherwise, the task or the DR operation may fail.
• During the configuration or deletion of the DR system, DR switchover, or forcible product data synchronization, the database at the secondary site is restarted. As a result, the management plane at the secondary site reports GaussDB T V3 process has not started. After the DR operation is complete, the alarm is automatically cleared.

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. In the upper right corner, click Delete.
3. Perform operations as prompted.

   If the heartbeat status between the primary and secondary sites is , the DR relationship of both sites will be deleted. If the heartbeat status is , the DR relationship of the peer site cannot be deleted, and therefore you need to perform the operation at both sites.

4. Check the operation result. If the operation result is not as expected, contact Huawei technical support.
   1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
   2. On the Manage Remote DR System page, verify that the DR system that has been deleted no longer exists.
   3. Verify that you can log in to the service plane at the active site. For details, see Logging In to service plane.
   4. On the Task List page, if the task details indicate that the DR system and product deletion is partially successful, the DR system may be deleted when a product node is powered off or abnormal. In this case, you need to manually clear the DR system information on the product node after the node is restored. Otherwise, the services on the node will be abnormal in a non-DR scenario. For details, see "Clearing the DR Information of the Product Nodes After the DR System Is Deleted" in Troubleshooting Guide.

5. Back up the product data and the management plane. This is because after the DR system is deleted, the historical backup files have become invalid. For details, see Backing Up Products and Backing Up the Management Plane.

   If you need to create a DR system again, back up the product data and the management plane after the DR system is created.

Follow-up Procedure

After the DR system is deleted, the product services at the primary and secondary sites are still in the state before the deletion.

• In the Distributed scenario, do not start services at the secondary site to prevent service preemption and repeated service provisioning caused by the dual-active state of products at the primary and secondary sites. If you need to start services at the secondary site, do not provision services at the primary and secondary sites during the startup of the secondary site.
• In the centralized scenario, if the floating IP address for northbound interconnection is configured for the primary and secondary sites and the services of the products at the secondary site are manually started, the floating IP address for northbound interconnection may conflict. Manually disable the NIC of the floating IP address for northbound interconnection at the secondary site.
  1. Use PuTTY to log in to the management node at the secondary site as the sopuser user in SSH mode.
  2. Run the following command to switch to the root user:

     > su - root

     Password: password for the root user

  3. Run the following command to query the NIC usage of the management node:

     # ifconfig

  4. If the query result contains the NIC for the floating IP address for connecting to the northbound network (for example, bond0:0), the NIC is not disabled. Run the following command to disable the NIC:

     # ifconfig bond0:0 down

  5. Run the following command again to query the NIC usage of the management node. The query result does not contain the NIC for the floating IP address for connecting to the northbound network.

     # ifconfig

Prerequisites

• The planning requirements for the primary and secondary sites are met. For details, see Checking Before Configuring the DR System.
• You have logged in to the management plane at the primary or secondary site. For details, see Logging In to the Management Plane.
• All services and database instances at the primary and secondary sites are running properly. For details, see System Monitoring.
• Ensure that the heartbeat status between the primary and secondary sites is .

Precautions

• Perform operations provided in this section only at one site.
• The database status needs to be updated after the DR system is deleted. You are advised to create a DR system 5 minutes after the old one is deleted. Otherwise, the operation may fail.
• If a backup or restoration task is in progress, perform operations in this section after the task is complete. Otherwise, the task or the DR operation may fail.
• In the automatic switchover (with the arbitration service) scenario, after the DR relationship between the primary and secondary site products is deleted, the product services and databases at the primary and secondary sites are still in the active or standby state before the deletion. To prevent the dual-active problem caused by the conflict between the status of the new product and that of the product before the deletion, disable the automatic switchover function before re-establishing the DR relationship between the primary and secondary site products. After a product is added, enable the automatic switchover function again.

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. In the automatic switchover (with the arbitration service) scenario, you need to disable the automatic switchover function.
   1. On the Manage Remote DR System page, click Configure DR System.
   2. Click the DR Extended Configuration tab, record the configuration information, and set Automatic Switch to .

3. On the Manage Remote DR System page, click Add Product.
4. Perform operations as prompted.
5. Check the operation result.
   1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
   2. On the Manage Remote DR System page, verify that the heartbeat status between the primary and secondary sites is .
   3. On the Manage Remote DR System page, verify that Data Synchronization Status of all products is Synchronized or Synchronizing. If Data Synchronization Status is Delayed, a large volume of data is being synchronized between the primary and secondary sites. Check the status after data synchronization is complete.
   4. Verify that you can log in to the service plane at the active site. For details, see Logging In to service plane.
   5. If the operation result is not as expected, contact Huawei technical support.

6. In the automatic switchover (with the arbitration service) scenario, click the DR Extended Configuration tab and set Automatic Switch to to enable the Automatic Switchover function. Restore the DR extension configuration by referring to 2.b.

Prerequisites

You have logged in to the management plane of the primary or secondary site. For details, see Logging In to the Management Plane.

Precautions

• After the primary and secondary site products are separated, products at the primary and secondary sites are independent from each other. The secondary site product no longer provides DR protection for the primary site product and data is not synchronized between the two sites. However, the product data at the two sites is not deleted.
• If a backup or restoration task is in progress, perform operations in this section after the task is complete. Otherwise, the task or the DR operation may fail.

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. In the row of the product with DR relationship to be deleted, click .
3. Perform operations as prompted.

   If the heartbeat status between the primary and secondary sites is , the operation at a site will synchronously delete the product from the DR system at the peer site. If the heartbeat status is , the operation cannot delete the product from the DR system at the peer site, and therefore you need to perform the operation at both sites.

4. Check the operation result. If the operation result is not as expected, contact Huawei technical support.
   1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
   2. On the Manage Remote DR System page, verify that the heartbeat status between the primary and secondary sites is .
   3. On the Manage Remote DR System page, check that the deleted product does not exist.
   4. Verify that you can log in to the service plane of the active site. For details, see Logging In to service plane.

Follow-up Procedure

To prevent the dual-active state, after the primary and secondary site products are separated, the product services and databases remain in the state before the deletion. If you need to start the services at the secondary site, log in to and perform the operations at the secondary site. For details, see Starting the Service Plane.

Prerequisites

• You have logged in to the management plane of the primary or secondary site. For details, see Logging In to the Management Plane.
• The heartbeat status between the primary and secondary sites is .

Precautions

• Perform operations provided in this section only at one site.
• After the data synchronization relationship between the primary and secondary site products is deleted, the data at the two sites still exists, but the data, excluding the HFS files, cannot be synchronized. To delete the synchronization relationship between the HFS files data at the primary and secondary sites, you need to separate the primary and secondary site products or delete the DR system. For details, see Separating the Primary and Secondary Site Products and Deleting the DR System.
• If a backup or restoration task is in progress, perform operations in this section after the task is complete. Otherwise, the task or the DR operation may fail.

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. On the Manage Remote DR System page, verify that Data Synchronization Status of all products is Synchronized or Synchronizing. If Data Synchronization Status is Delayed, a large volume of data is being synchronized between the primary and secondary sites. Check the status after data synchronization is complete.
3. In the row of the product with data synchronization relationship to be deleted, click .
4. Perform operations as prompted.
5. Check the operation result. If the operation result is not as expected, contact Huawei technical support.
   1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
   2. On the Manage Remote DR System page, verify that the heartbeat status between the primary and secondary sites is .
   3. On the Manage Remote DR System page, verify that Data Synchronization Status of the products is Abnormal.
   4. Verify that you can log in to the service plane of the active site. For details, see Logging In to service plane.

Follow-up Procedure

After the fault of the primary site product is rectified, restore the data synchronization relationship between the primary and secondary site products based on site requirements.

• If data generated on the products at the primary site does not affect services, fully synchronize data from the primary site product to the secondary site product. For details, see Synchronizing Product Data Between Primary and Secondary Sites.
• If data generated on the products at the primary site affects services, fully synchronize data from the secondary site product to the primary site product. For details, see Synchronizing Product Data Between Primary and Secondary Sites. After the synchronization, restore the products at the two sites to the status before the faults. For details, see Performing DR System Drills.

Prerequisites

• You have logged in to management plane of the primary or secondary site. For details, see Starting the Management Plane.
• The heartbeat status between the primary and secondary sites is , and the data synchronization status of all products is Synchronized or Synchronizing.

Precautions

• Perform operations provided in this section only at one site.
• During the switchover, if the data synchronization status is Synchronizing, data of the product is being synchronized between the primary and secondary sites. The system waits for the data synchronization to complete and then performs the switchover. If the waiting times out, the switchover fails, that is, the primary site still functions as the active site and the secondary site functions as the standby site.
• If a backup or restoration task is in progress, perform operations in this section after the task is complete. Otherwise, the task or the DR operation may fail.
• During the configuration or deletion of the DR system, DR switchover, or forcible product data synchronization, the database at the secondary site is restarted. As a result, the management plane at the secondary site reports GaussDB T V3 process has not started. After the DR operation is complete, the alarm is automatically cleared.

Context

After switchover, the status of products at both sites will change. As shown in Figure 3-27, after the product at the primary site is successfully switched over to the product at the secondary site, the product at the primary site becomes standby and the product at the secondary site becomes active. The data synchronization direction of products between the primary and secondary sites also changes accordingly.

Figure 3-27 Diagram of site status before and after switchover

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. On the Manage DR System page, perform operations as required.
   • DR system drill for a single product

     In the Operation column of the row that contains the product, click and perform operations as prompted.

   • DR system drill for all products

     Select all products, click Switch Over above the product list, and perform operations as prompted.

   During the data synchronization, the database status on the System Monitoring page of the Standby site may be Abnormal. After the data synchronization is complete, the database status changes to Normal.

3. Check the operation result. If the operation result is not as expected, contact Huawei technical support.
   1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
   2. On the Manage DR System page, verify that the heartbeat status between the primary and secondary sites is .
   3. On the Manage DR System page, view information in the Primary Site Product and Secondary Site Product columns and verify that the product DR status is consistent with the switchover result.
   4. On the Manage DR System page, verify that Data Synchronization Status of the switched products is Synchronized or Synchronizing. If Data Synchronization Status is Delayed, a large volume of data is being synchronized between the primary and secondary sites. Check the status after data synchronization is complete.
   5. Verify that you can log in to the service plane at the active site and the menus can be displayed properly. For details, see Logging In to service plane.

4. Perform a switchover again to restore the status of the original active and standby sites.
5. (Optional) Manually start processes or change the process startup type based on site requirements.
   • If Startup Type of a process is Manual, perform the following operations:
     1. On the management plane, choose Product > System Monitoring from the main menu.
     2. In the upper left corner of the System Monitoring page, move the pointer to and select the product.
     3. On the Processes tab page, select the process to be started, click Start in the upper right corner of the process list, and perform operations as prompted.
   • For details about how to change the startup type of a process, see Configuring Process Startup Types.

Prerequisites

• You have logged in to the management plane. For details, see Logging In to the Management Plane.
• The communication of the heartbeat links and data replication link between the primary and secondary sites is normal.

Context

• After the DR system is created, a scheduled task for the health check is automatically created on the management plane of the secondary site and is executed at 07:00:00 every day to perform the health check. For details about how to change the execution time of the scheduled task, see Modifying DR System Parameters.
• If a check item is abnormal, the DR system checks all items every hour until they are normal or until the next scheduled health check starts.

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. On the Manage Remote DR System page, click Evaluate Health and perform operations as prompted.
3. Rectify the items whose Result is Abnormal based on the suggestions.

Prerequisites

You have logged in to the management plane. For details, see Logging In to the Management Plane.

Precautions

If you need to change the name or heartbeat IP address of the primary and secondary sites, or change the product deployed at the two sites, you need to delete the existing DR system and create a DR system again. For details, see Deleting the DR System and Configuring the DR System (Manual Switchover).

Procedure

1. On the management plane at the primary site, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. On the Manage Remote DR System page, click Edit and perform operations as prompted.

Prerequisites

You have logged in to the management plane of the secondary site. For details, see Logging In to the Management Plane.

Context

After the takeover, the product status at both sites changes. As shown in Figure 3-28, the primary site product is faulty, and the secondary site product takes over services from the primary site product, and the secondary site product changes to active from standby.

Figure 3-28 Before and after the primary site product services are taken over

Precautions

• Service takeover can be performed only on products in the Standby state. After the takeover is successful, products at the standby site take over services of the products at the active site and provide services externally.
• During the takeover, if the data synchronization status is not Synchronized, the takeover may cause product data loss.

Procedure

1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
2. On the Manage Remote DR System page, perform operations as required.
   • If a single product is faulty:

     In the Operation column of the row that contains the product, click and perform operations as prompted.

   • If multiple products are faulty:

     Select the products to be taken over, click Take Over above the product list, and perform operations as prompted.

3. Check the operation result. If the operation result is not as expected, contact Huawei technical support.

   If the heartbeat status between the primary and secondary sites is abnormal, or the primary site product fails to become standby during the takeover, the DR system enters the dual-active state, which may cause data loss. After the heartbeat status between the primary and secondary sites is restored, you need to forcibly synchronize data between the products at the primary and secondary sites to ensure data consistency between the products at the primary and secondary sites. For details, see Synchronizing Product Data Between Primary and Secondary Sites.

   1. On the management plane, choose HA > Remote High Availability System > Manage DR System from the main menu.
   2. On the Manage Remote DR System page, check that the DR status of the product that has taken over services in 2 is Active or Active after takeover.
   3. Verify that you can log in to the service plane of the new active site. For details, see Logging In to service plane.

4. (Optional) Manually start processes or change the process startup type based on site requirements.
   • If Startup Type of a process is Manual, perform the following operations:
     1. On the management plane, choose Product > System Monitoring from the main menu.
     2. In the upper left corner of the System Monitoring page, move the pointer to and select the product.
     3. On the Processes tab page, select the process to be started, click Start in the upper right corner of the process list, and perform operations as prompted.
   • For details about how to change the startup type of a process, see Configuring Process Startup Types.

Follow-up Procedure

After the takeover is complete, restore the DR system according to the system status. For details, see Table 3-121.

Fault Symptoms

On the Manage Remote DR System page of the management plane, the heartbeat status is (Abnormal) or (Unknown).

Figure 3-29 DR heartbeat status visualization page

Figure 3-30 shows the DR heartbeat list. You can click to expand the faulty heartbeat list and view the faulty node.

Figure 3-30 DR heartbeat list

Symptom 1: Abnormal Arbitration Heartbeat Between the Primary and Secondary Sites

Symptom 2: Abnormal Arbitration Heartbeat Between the Primary and Third-Party Sites

Symptom 3: Abnormal Arbitration Heartbeat Between the Secondary Site and Third-Party Site

Symptom 4: Unknown Arbitration Heartbeat Between the Primary/Secondary Site and Third-Party Site

Symptom

On the Manage DR System page of the management plane, the heartbeat status between the primary and secondary sites is (abnormal).

Possible Causes

• The heartbeat network between the primary and secondary sites is abnormal.
• The DR service at the primary or secondary site is abnormal.
• The DR system certificates of the management node at the primary and secondary sites are inconsistent or have expired.

Prerequisites

• You have obtained the heartbeat IP address of the management node at the secondary site.
• You have obtained the password for the sopuser and ossadm user on the management node at the primary and secondary sites.

Troubleshooting Procedure

This section provides only the basic troubleshooting methods. If the fault persists after troubleshooting using the following methods, contact Huawei technical support.

1. Check whether the heartbeat network between the primary site and secondary site is normal.
   1. Use PuTTY to log in to the management node at the primary site as the sopuser user in SSH mode. Run the following command to switch to the ossadm user:
      > su - ossadm

      Password: password for the ossadm user

      If the management plane is deployed in cluster mode, perform operations only on OMP_01. For details about how to obtain the IP address of a node, see How Do I Query the IP Address of a Node?

   2. Run the following command to test the connectivity between the management nodes at the primary and secondary sites.

      • If the IP address version is IPv4:

        > ping heartbeat IP address of the management node at the secondary site

      • If the IP address version is IPv6:

        > ping6 heartbeat IP address of the management node at the secondary site

      Check the command output.

      • If information similar to the following is displayed, the IP address can be pinged, and the network connection is normal:

        64 bytes from heartbeat IP address of the management node at the secondary site: icmp_seq=1 ttl=251 time=42.1 ms

      • If no command output is displayed within 1 minute, the network connection is abnormal. Contact the administrator to check the network status and rectify the network fault.
   3. Press Ctrl+C to stop the ping command.

2. Check whether the DR processes of the management node are normal at the primary and secondary sites.
   1. Log in to the management plane at the primary site. For details, see Logging In to the Management Plane.
   2. On the management plane choose Product > System Monitoring from the main menu.
   3. In the upper left corner of the System Monitoring page, move the pointer to and select iMaster NCE-Campus-OMP.
   4. On the Services tab page, click UniEPMgr.
   5. In the Processes area, check whether the drmgrservice-x-x process exists and whether Status of the process is Running.

      x indicates the instance number. Replace it based on site requirements.

      • If yes, the processes exist and are running properly.
      • If no, contact Huawei technical support.
   6. Log in to the management plane at the secondary site, and perform the preceding operations to check the DR processes at the secondary site. If abnormal, contact Huawei technical support to restore the DR processes.

3. Check whether the DR system certificate of the management node at the primary and secondary sites has expired.
   Check whether the 51025 Certificate of the Remote DR System Has Expired alarm is generated for the primary and secondary sites.

   • If yes, update the DR system certificate. For details, see "Updating DR System Certificates" in Maintenance and Monitor (Management plane).
   • If no, this fault is not caused by DR system certificate expiration.

4. Contact Huawei technical support to check whether the DR system certificates of the management node match between the primary and secondary sites.