No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

High CPU usage caused by 'trace object' function enabled.

Publication Date:  2012-07-27 Views:  57 Downloads:  0
Issue Description
Due to radius server was irresponsible and me60 (v100r006c05spc600) was retransmitting great number of accounting packets - the me60 cpu usage get up to 100%. The service, telnet access to device was interrupted and console access was very slow. After some time, when radius server recovered - the cpu usage decreased and remote access to me60 restored.
In normal situation radius server problem shouldn't influence me60 in such way.

Alarm Information
The 'display cpu-usage' command output shows that rds process occupy cpu greatly:
me60_1>
[me60-1]display cpu-usage
cpu usage stat. cycle: 60 (second)
cpu usage : 100% max: 100%
taskname cpu runtime(cpu tick high/cpu tick low)
rds 52% 0/3e6d1d10

Handling Process
1. Check if 'trace object' function is enabled in hide-view:

[me60-hidecmd] display trace information
-------------------btrc trace infomation--------------------
trace enable:                                              <--- THIS MEANS TRACE IS ENABLED (0 - DISABLED)

the sum of all the instanses is : 25512   <--- GREATE NUMBER OF TRACE INSTANCES WILL OCCUPY CPU
the startid of the instanse table is: 15

the sum of all the objects is : 2
the detail information :
index objectid
0 the id of this object is : 0
1 the id of this object is : 1


2. Disable the trace object function:

undo trace object all
undo trace enable



Root Cause
1. From present view, cpu high caused by many accounting packets had been resent because link quality was bad between radius server and me60
(there were lots of log info: "oct 12 2011 23:58:45 oren_ats-56_me60_1 %%01vsm/4/rd_acct_rtfail(l):-slot=0; normal real time account for oren_at13254ssg0246654ccf75159425 failed, please check radius accounting server")
2. The reason above couldn't load the CPU till 100% by itself.
3. The RDS process is also responsible for printing trace information for monitored objects (subscribers).
4. Check if 'trace object' function is enabled and trace instances number is great.

Suggestions
Due to trace switch always open, and plenty number of user applied trace instance, therefore radius module is often transfer trace function to print info, so when subscribers frequently get online and offline, radius module occupy rate will be very high. Trace function is just debugging method, and if it will be used after problem happens, cpu usage will be occupied much more if long term open. According to operation criterion, trace function should be closed after using finished.

 



END