There are two HP RX6600 midrange computers, and an OceanStor S3100 storage takes as the server of the Oracle database. The midrange computer has installed the MC/Service Guard two-node cluster software, and the version is: A.11.17.00.
1. Scan the device file via the “ioscan”, we can see the hardware device of the storage normally.
2. Check the two-node cluster logs, we find there is monitoring error information, as the followed displayed:
TNSLSNR for HPUX: Version 188.8.131.52.0 - Production
System parameter file is /oracle/product/9.2/network/admin/listener.ora
Log messages written to /oracle/product/9.2/network/log/listener.log
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=icdnode1)(PORT=1521)))
TNS-12542: TNS:address already in use
TNS-12560: TNS:protocol adapter error
TNS-00512: Address already in use
HPUX Error: 226: Address already in use
Listener failed to start. See the error message(s) above...
ERROR: Function customer_defined_run_cmds
ERROR: Failed to RUN customer commands
Oct 16 13:29:09 - Node "icdnode1": Halting service ORAMONITOR
Analyze: while the two-node cluster resource restarting, there is a Oracle monitoring process is running, which causes the new process running failed, and then the whole two-node cluster resource been hang up, before running the Oracle monitoring process, the two-node cluster resource runs normally, the VG status is UP; after the Oracle monitoring has run failed, the whole two-node cluster resource has been hang up, and the VG status changes to be Down.
3.Stop the Oracle monitoring process, and make sure there isn’t any Oracle monitoring process while the two-node cluster is starting, and then run the two-node cluster, all are normal.
The steps to stop the Oracle monitoring process are as followed:
/opt/oracle> sqlplus "/as sysdba"
/opt/oracle >lsnrctl stop
View via the “vgdisplay” after the two-node cluster has started, the vg resource has changed to be down status after stayed in the up status transitorily, and then the two-node cluster changes to be down status and it can’t start normally. Check the storage connection is normal, the disk device file of the HP-UX is normal and we haven’t found any abnormal problem.
1. The script of the two-node cluster needs to be improving and perfect, while the starting the Oracle monitoring, validate whether the monitoring is running, and then implement the next step.
2. For the software and system problem, read the logs carefully and repeatedly, it’s helpful to locate the problem.