No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Performance impact on host CCMS due to low R/W IOPS response

Publication Date:  2017-10-30 Views:  32 Downloads:  0
Issue Description

We faced the performance issue on CCMS host on Monday (30th Jan ) morning after 6:30 AM and that time we analysed MIS host  was taking high
bandwidth and IOPS, Means ggnccm01 and ggnccm02 host not able to provide IOPS to application/database resulted in process got hung.

 




 



<1.Performance data analyse>

Performance data is UTC+8 time zone



 

HOST NAME


 

 

HOST ID


 

 

CCMS01


 

 

8


 

 

CCMS02


 

 

9


 


 


  
 


 

MIS01


 

 

6


 

 

MIS03 


 

 

7


 



 

Handling Process

1.1 On UTC+8 9am local time 6:30 amhost MIS01 and MIS02 had low IOPS and Bandwidth, Bandwidth is below
50MB/s
MAX IOPS below 20K

 



Bandwidth



 



1.2     On UTC+8 9am local time 6:30 amhost8(CCMS01) had stable IOPS and Bandwidth, but host9(CCMS02) IOPS had obvious increase from IOPS 4000 to 30000.

 

bandwidth



<2.Check storage event log>

2.1  No any relevant error event during the issue happen.

2.2  Avg IO latency of all four host is below 5ms, and did not had any obvious change

2.3  We did not find any IO latency timeout error event.

3<Analyze Storage Configuration>

In storage side, totally 8* FC port is in linkup status. All the port speed is negotiation as 4Gb/s.

 

ID: ENG0.A1.P3 Working Rate(Mbps): 4000

ID: ENG0.A1.P3 Working Rate(Mbps): 4000

ID: ENG0.A2.P1 Working Rate(Mbps): 4000

ID: ENG0.A2.P3 Working Rate(Mbps): 4000

ID: ENG0.B1.P1 Working Rate(Mbps): 4000

ID: ENG0.B1.P3 Working Rate(Mbps): 4000

ID: ENG0.B2.P1 Working Rate(Mbps): 4000

ID: ENG0.B2.P3 Working Rate(Mbps): 4000

 Each 4Gb FC port transfer rate about 375MB/s, each host had config two FC port, so the max transfer rate of one host is about 750MB/s

Host CCMS_BKP_HOST  (host id :10)  had very high transfer rate about 750MB/S when the issue happen.



 

Root Cause


The Performance data is UTC+8 time zone and the screenshot as below is UTC+8 time zone.

From the performance log , the host6 (MIS01) IOPS  reached 12K from 12:30(UTC +8) , so please check if the active is on 10:00 local time ?

During that time , the host6 read bandwidth is keeping 780MBps and it reach the bottleneck of host link.   host6(MIS01) and host89(CCMS)share the physical link  and storage port


we found that MIS01/03 and CCMS01/02 was shared the physical storage ports so this  impact the CCMS host performance.



Solution

In existing environment we  found that CCMS and MIS both application were using the same Physical storage front end port

We did the Separate FC port Zoning for both CCMS and MIS application after that issue resolved.

 

Suggestions

in Heterogeneous environment please do the seprate Zoning for Highly utilized application or enable Smart Quota Feature as per the application requirement.

END