No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

How to Respond to Threshold-Crossing Memory usage of Cache Logs on an NE40E

Publication Date:  2019-04-22 Views:  13 Downloads:  0
Issue Description

An NE40E X8A V800R008C10SPC100 installed with the patch V800R008SPH007 is restarted. The analysis of the collected information shows that the restart is caused by threshold-crossing memory usage of the cache logs. Check the memory usage of the cache logs three days after the fault occurs. It is found that the memory usage increases to 77%. High memory usage of the cache logs may cause device restart.

Alarm Information

In the display device command output, Slot10 indicates the master main control board and Slot9 indicates the backup main control board. In the display diaglog slot 9 disk_usage index 1 command output (used to view the memory usage of the related files on main control boards), the memory usage of the cache log of the master main control board is 77% and that of the backup main control board is 3%.

<NE40E-BR-02>dis dev

NE40E-X8A's Device status:

-------------------------------------------------------------------------------

Slot #   Type    Online     Register     Status     Role   LsId   Primary      

-------------------------------------------------------------------------------       

9        MPU     Present    Registered   Normal     MMB    0      Slave        

10       MPU     Present    Registered   Normal     MMB    0      Master        

-------------------------------------------------------------------------------


[~NE40E-BR-02-diagnose]display diaglog slot 9 disk_usage index 1

Filesystem                Size      Used Available Use% Mounted on

/dev/sda4               121.5M      5.5M    109.7M   5% /opt/vrpv8/nextpatch

/dev/mtdblock9           30.0M    848.0K     29.2M   3% /opt/vrpv8/var


[~NE40E-BR-02-diagnose]display diaglog slot 10 disk_usage index 1

Filesystem                Size      Used Available Use% Mounted on

/dev/sda4               121.5M      5.5M    109.7M   5% /opt/vrpv8/nextpatch

/dev/mtdblock9           30.0M     23.1M      6.9M  77% /opt/vrpv8/var

Handling Process

1. Run the display switchover state command to check whether the device supports master/backup main control board switching. Such switching can be performed only when the switchover state is Ready.

2. Run the slave switchover command to perform a master/backup main control board switchover. In this case, the main control board in slot 9 changes to be the master and the main control board in slot 10 changes to be in the Unregistered state.

3. Run the display device and display switchover state commands to check whether the main control board in slot 10 is registered and whether the conditions for master/backup main control board switching are met.

4. Run the display diaglog slot 9 disk_usage index 1 command to check whether the memory usage of the cache logs on the main control board is normal.


<NE40E-BR-02>display switchover state 

Chassis              :  Single

   Switchover State  :  Ready

   Switchover Policy :  Board Switchover

   MainBoard         :  10

   SlaveBoard        :  9


[NE40E-BR-02]slave switchover         

Warning: This operation will switch the slave board to the master board. Continue? [Y/N]:y

Please wait.

Switching OK

 This board has been switched slave successfully, and it will be reset!


<NE40E-BR-02>dis dev

NE40E-X8A's Device status:

-------------------------------------------------------------------------------

Slot #   Type    Online     Register     Status     Role   LsId   Primary      

-------------------------------------------------------------------------------     

9        MPU     Present    Registered   Normal     MMB    0      Master       

10       MPU     Present    Unregistered Abnormal   MMB    0      NA           

-------------------------------------------------------------------------------

<NE40E-BR-02>dis switchover state 

Chassis              :  Single

   Switchover State  :  No slave

   Switchover Policy :  Board Switchover

   MainBoard         :  9

   SlaveBoard        :  10

<NE40E-BR-02>dis dev              

NE40E-X8A's Device status:

-------------------------------------------------------------------------------

Slot #   Type    Online     Register     Status     Role   LsId   Primary      

-------------------------------------------------------------------------------      

9        MPU     Present    Registered   Normal     MMB    0      Master       

10       MPU     Present    Registered   Normal     MMB    0      Slave        

-------------------------------------------------------------------------------

<NE40E-BR-02>dis switchover state 

Chassis              :  Single

   Switchover State  :  Ready

   Switchover Policy :  Board Switchover

   MainBoard         :  9

   SlaveBoard        :  10


[~NE40E-BR-02-diagnose]display diaglog slot 9 disk_usage index 1

Filesystem                Size      Used Available Use% Mounted on

/dev/sda4               121.5M      5.5M    109.7M   5% /opt/vrpv8/nextpatch

/dev/mtdblock9           30.0M      1.2M     28.8M   4% /opt/vrpv8/var


[~NE40E-BR-02-diagnose]display diaglog slot 10 disk_usage index 1

Filesystem                Size      Used Available Use% Mounted on

/dev/sda4               121.5M      5.5M    109.7M   5% /opt/vrpv8/nextpatch

/dev/mtdblock9           30.0M    848.0K     29.2M   3% /opt/vrpv8/var

Root Cause

The customer has changed the NMS platform from Telnet to SSH. The number of device accesses by SSH is large and the frequency is high, causing the memory usage of cache logs to increase. The NE40E X8A is restarted when the memory usage of the cache logs reaches the threshold.

Solution

Pay special attention when the memory usage of the cache logs reaches 80%. If the memory usage keeps on 85% in multiple queries, perform a master/backup main control board switchover to prevent device restart. Then, troubleshoot the problem.

Suggestions

If the memory usage of an NE40E exceeds the specified threshold and the NE40E is equipped with two main control boards, perform a master/backup main control board switchover to prevent device restart. The switchover operation does not affect service forwarding.

END