Oracle Service Failed to Start After the FusionCube Database Infrastructure System Is Power Cycled

Publication Date:  2014-06-06 Views:  263 Downloads:  0
Issue Description
 The Oracle service of some database nodes failed to start after the storage nodes and database nodes of the FusionCube Database Infrastructure system are power cycled.
Alarm Information
None
Handling Process
1. Use PuTTY to log in to the DB node where the Oracle service fails to start.
2. Run the following command to query the ASM disks of the Oracle database:
/usr/sbin/oracleasm listdisks
Information similar to the following is displayed:
[root@dsware1 ~]# /usr/sbin/oracleasm listdisks
DISK1
DISK10
DISK2
DISK3
DISK4
DISK5
DISK6
DISK7
DISK8
DISK9
[root@dsware1 ~]#
3. Check whether the ASM disks queried in step 2 are consistent with the configured ASM disks.
 If yes, contact Huawei technical support.
 If no, go to step 4.
4. Run the following command to scan for the ASM disks of the Oracle database again:
/usr/sbin/oracleasm scandisks
5. Run the following command to query the ASM disks of the Oracle database again:
/usr/sbin/oracleasm listdisks
6. Check whether the ASM disks queried in step 5 are consistent with the configured ASM disks.
 If yes, go to step 7.
 If no, ensure that information about the attached volumes is correct and go to step 4.
You can run the following command to check information about the attached volumes:
/opt/dsware/agent/tool/lsscsi
7. Check whether the Oracle service starts.
 If yes, no further action is required.
 If no, manually restart the Oracle service or restart the server and check the Oracle service again. If the fault still cannot be rectified, contact Huawei technical support.
Root Cause
When the whole FusionCube database infrastructure system is power cycled, the startup sequence of the Object Storage Device (OSD) and Virtual Block System (VBS) processes in a storage cluster is in random order. Some storage nodes may take a long time to start. Therefore, data of some volumes on the storage nodes is not ready for use. After the VBS process successfully attaches volumes to a node, the oracleasm service of the Oracle database will automatically restarts to enable the system to scan for ASM disks again. However, if data of some volumes is not ready, the system fails to detect the ASM disks. As a result, the Oracle service is faulty and fails to start. 
Suggestions
None

END