No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Two Active MDS Services Are Detected Due to Inconsistent Subnet Masks of Heartbeat Network Ports

Publication Date:  2014-09-23 Views:  97 Downloads:  0
Issue Description
  • Product involved: CSS-F
  • Product version: CSS-F V001R001C01

After an MDS is deleted and added to the CSS again, two active MDS services are detected. Then the heartbeat service on MDS1 is stopped and MDS2 becomes the active MDS).
Alarm Information
Two active MDS services are detected in the cloud storage domain.
Heartbeat services on both active and standby MDSs are active, causing one MDS to be stopped.
Handling Process

1.Modify the subnet mask of the heartbeat network port on MDS1 to 255.255.0.0 (consistent with that of the heartbeat port on the active MDS). During the modification, do not restart the network. 

               (1)Run the ifconfig bond1 192.168.0.x netmask 255.255.0.0 command.
               (2)Modify the configuration file.
                    Run the vi /etc/sysconfig/network/ifcfg-bond1 to modify the subnet mask.
                    NETMASK='255.255.0.0'
2.Wait one minute and check the HA status.
Run the mdsmml mds hastatus command on MDS1 and MDS2 respectively.
The state of the active MDS is primary MDS and the state of the standby MDS is secondary MDS.


 

Root Cause
  • After HA logs are analyzed, it is found that this fault is caused by a heartbeat software error. The heartbeat between the active and standby MDSs is normal and heartbeat IP addresses can be pinged. However, the software mistakes that the heartbeat is down.
  • UDP broadcast packets are sent between the two heartbeat network ports to check the heartbeat status.
  • The subnet masks of the heartbeat network ports are inconsistent. When MDS1 is re-added to the CSS after being stopped, the subnet mask for the heartbeat port on MDS1 is set to 255.255.255.0, inconsistent with that used by the heartbeat port on MDS2. The subnet mask used by the heartbeat port on MDS2 is 255.255.0.0. After MDS1 is added to the CSS, the IP address/subnet mask for MDS1 and MDS2 are 192.168.0.x/24 and 192.168.0.y/16 respectively.
  • The ICMP ping succeeds but the UDP ping fails, which causes the standby MDS to be switched over to the active MDS. However, the active MDS in use is still working. After the HA monitoring script detects two active MDSs, the script stops the active MDS in use.
Inconsistent subnet masks of the heartbeat network port on the MDSs cause the communication breakdown between the heartbeat network ports. As a result, two active MDSs are detected. 
Suggestions
When deploying or expanding the CSS, pay special attention to the subnet masks of heartbeat network ports on the active and standby MDSs. Ensure that the subnet masks are consistent.

END