Memory utilization rate reach 100 occured in S5500TV2 when 10GE TOE card failure

Publication Date:  2016-03-06 Views:  355 Downloads:  0
Issue Description

S5500T, system version is V200R002C00SPC400

There are 10GE TOE cards in the storage, BOM code is 0302G367

 

 

Alarm Information

There're historical events like below. We can see a lot of 10GE TOE card B0 was succeeded in Powering on events at first.

2016-02-22 21:26:13    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 20:55:45    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 20:43:15    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 20:25:23    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 19:54:59    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 19:24:38    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 18:54:11    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).
2016-02-22 17:39:31    0x200f0d1000f    Infor    None    System succeeded in powering on the interface module (engine ENG0, 2 interface module B0).

 

After a few hours, the system reported memory usage is too high, as below:

 2016-02-23 09:59:40    0xf0c90005    Warning    2016-02-23 10:00:41    The memory utilization rate threshold (100)of the controller B in the enclosure Engine 0 has been reached.
2016-02-23 09:57:40    0xf0c90005    Warning    2016-02-23 09:58:41    The memory utilization rate threshold (100)of the controller B in the enclosure Engine 0 has been reached.

Later, the controller reset, alarm as below:

2016-02-23 22:05:49    0xf00cf0014    Critical    2016-02-23 22:12:48    The controller module (Engine ENG0,id B) can not be monitored. The error code is --.
2016-02-23 22:05:49    0x100f00cf0034    Critical    None    The controller (2) {0: controller enclosure; 1: disk enclosure; 2: engine} ENG0, controller B) is isolated.The error code is 0x40401703.
2016-02-23 22:05:49    0xf0cf0005    Major    2016-02-23 22:05:50    The communication between the controller (A) and the other controller (B) is abnormal in the enclosure (Engine ENG0), but the system can continue to work with error code 0x4000cf12.

Handling Process

1.Collect storage logs, then analyze why controller B was abnormal and isolated. Unzip logs of controller B as below. Only analyze files with prefix "log_debug_" and "log_reset_", each file include the timepoint that the file was generated. The files with prefix "log_reset_" record the reset reason at that time. The files with prefix "log_debug_" record debug logs before and after the reset. So, we just analyze "log_reset_20160223221139_poweron.txt" and "log_debug_20160223221139_poweron.txt" in this case.

2.First, we can find the reset reason is Out of memory reset in "log_reset_20160223221139_poweron.txt". It means that system memory was used up and storage reset controller B  to self-heal.

The latest NO.1 reset: localorcmostime=1456261538, ji=5817415460, reason=out of memory reset
Desktime=2016-02-23-22:05:38

3.Second, we can search critical word "BIOS-p" to find the  logs after reset, then we can look back to analyze why memory was used up.

Then ,we can see the list of files which stored in memory, also we have file size in memory. So, we can find these files which occupied extra memory space before Out of memory reset as below.

Then, we have see the output of bash command "ps -aux" as below. We can find these threads who occupied extra memory space.The RSS item represent the physical memory space of the coresponding thread.


Also, we can find memory trace information of  kernel moudules  like below. We can get the usage of memory for each kernel module.

4. In this case, we finally find many unexpected performance statistics file in "/OSM/script" directory as below.



We can calculate that they occupied about 300MB memory in total and this obviously caused the Out of memory reset.

Root Cause

1. The 10GE TOE card hard reset many times and this caused the performance statistics thread cored since it need to inquiry the performance data of 10GE cards.

2. Normally, the performance statistics thread records data file in memory directory(/OSM/script) first, then copy to coffer disk and delete old ones. Since the thread cored it didn't delete performance data files after copy to the coffer disk. So, there were a lot of historical performance file remain in memory, more and more. Finally, it used up memory of OS and caused controller reset.

Solution

1.Poweroff the faulty 10GE TOE card through ISM or CLI command line.  Then apply a spare part to replace it.

2.Contact Huawei support to clear the remain performance statistics files in memory directory.

END