No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Messages Log Dump Failure Due to Abnormal cron Process

Publication Date:  2014-09-23 Views:  132 Downloads:  0
Issue Description
•Product involved: CSS-F
•Product version: CSS-F V001R001C01

An alarm is reported on the ISM indicating insufficient log space.
After the du command is run to check the “/var” partition, it is found that more than 9 GB space of the total 11 GB portion has been used. After the size of each folder in the partition is checked, it is found that folder “/var/log” occupies large space. After the du -sm /var/log/* | sort -n is run to check details, it is found that the size of the messages log is 2.6 GB. After the latest bz2 package dump is checked, it is found that the messages log is not dumped for a long time. The log space is insufficient due to the messages log dump failure.

Alarm Information
The log space is to be fully occupied
Handling Process
1.Kill the abnormal cron process.

2.Manually restart the cron process.

The WCHAN state is - after the manual restart.

Root Cause
1.cron and syslog are restarted to rectify the log dump problem.
The problem may be caused by abnormal cron and syslog. First, the following command is run to restart the cron process:
service cron restart
Two days later, the problem persists. The following command is run to restart syslog:
service syslog restart
Two days later, the problem persists. Further actions are taken to diagnose the problem.

2.The following are analyzed:
  • Process status
The ps command with parameter 1 specified is run to obtain the detailed process status information.

The command output shows that WCHAN of the cron process is pipe_w. WCHAN is described as follows:
address of the kernel function where the process is sleeping (use wchan if you want the kernel function name). Running tasks will display a dash ('-') in this column.
The output shows that process cron is in the sleeping state and is waiting for processing read/write from subprocesses.
  • pipe_w status
[root@OSN70 14:54:38 /var/log]# ps -l|sort -k4|grep -v 'CMD'
4 S     0  7467  7459  0  76   0 -  2479 wait   pts/1    00:00:00 bash
0 R     0 27981  7467  0  77   0 -   837 -      pts/1    00:00:00 ps
0 S     0 27982  7467  0  78   0 - 13279 pipe_w pts/1    00:00:00 sort
0 S     0 27983  7467  0  79   0 -   756 pipe_w pts/1    00:00:00 grep

The preceding output shows that the parent process of processes ps, sort, and grep is bash. The bash process invokes subprocesses separated by vertical lines (|) in sequence. In this case, the sort and grep subprocesses are all in the waiting state.
  • Fault scenario
The cron process periodically executes tasks every day. When executing “/etc/crontab”, the cron invokes “test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1” first and then “/usr/lib/cron/run-crons”. Then “/etc/cron.daily/logrotate” is executed (The configuration file is under “/etc/logrotate.d/syslog”.) Finally, the messages log is renamed and dumped.

The cron process stays in state pipe_w (pipe-wait in full spelling). This process suspension is caused by the anomaly of a process writing the messages log during the log dump. During the messages log dump, process caupgrade opens and writes the log abnormally.















Suggestions
After the message log dump failure, first check the cron and syslog-gn processes. If any of the processes exit, restart the processes. If the processes fail to start correctly, check the status of cron process.
ps -el | grep cron
If the cron process is in the pipe_w state, perform the preceding steps to restart it.

END