Service Operations Fail Due to Abnormal MySQL Process on a System Data Node

Publication Date:  2012-07-17 Views:  177 Downloads:  0
Issue Description
Related information about the product and version: CSE V100R001C00.
Reading and writing data fail.

System data nodes are deployed on the T3200.
Alarm Information
Handling Process
Step 1     Run the /etc/init.d/mysql stop command on the system data node to stop the MySQL process.

                               Step 2     Run the /etc/init.d/mysql start command on the system data node to start the MySQL process.

                               Step 3     Run the ps ef|grep mysql command to check whether the MySQL process is running properly.

                               Step 4     Check whether reading and writing data can be performed successfully. If yes, the fault is removed. If no, contact technical support engineers.


Root Cause
1.         Log in to the system data node by using KVM. Then run the ping command on the Fabric plane (internal). The execution result shows that the network connection between this node and the ISM server is interrupted.
2.         Check the switches, T8000 server, and system data nodes. These devices are properly powered on. Network cables among devices are correctly connected. Status of network port indicators are normal. Therefore, service operation failure is not caused by hardware faults or abnormal network communication.
3.         Run the ps ef|grep mysql command on the failed system data node. The MySQL process is found disabled. If the MySQL process is enabled, the following information will be displayed after the execution of the command:
root      5688     1  0 Jul26 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/ mysql     6508  5688  4 Jul26 ?        02:02:40 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/DB-Above.err --open-files-limit=8192 --pid-file=/var/lib/mysql/ --socket=/var/lib/mysql/mysql.sock --port=3306 root     30655 24723  0 09:58 pts/1    00:00:00 grep mysql

4.         Run the ip addr command. The virtual IP address of the system data node is included in bond0. bond0 generally includes inet scope global bond0. x.x.x.x indicates the virtual IP address of the system data node.
Therefore, the service operation failure is caused by the abnormal MySQL process on this node.