No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Performance Monitoring Guide

OceanStor Dorado V3 Series V300R002

This document describes performance monitoring of storage systems, including the monitoring method, indicator planning, configuration monitoring, and problem diagnosis.
Rate and give feedback:
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Understanding the Performance of Front-End Ports

Understanding the Performance of Front-End Ports

Front-end ports process host I/Os. Learning about the relevant factors may help in the understanding and identification of potential bottlenecks which affect performance in a storage system.

Checking Information About Front-End Ports

Before analyzing the performance of front-end ports, check the locations of interface modules and the number, states, and speeds of connected ports.

You can query information about front-end ports using OceanStor DeviceManager or running the CLI command.

  • Use OceanStor DeviceManager to query information about front-end ports.

  • To query information about front-end ports, run show port general on the CLI.
    admin:/>show port general physical_type=FC 
     
      ID          Health Status  Running Status  Type       Working Rate(Mbps)     
      ----------  -------------  --------------  ---------  ------------------     
      CTE0.A5.P0  Normal         Link Up         Host Port  8000                   
      CTE0.A5.P1  Normal         Link Down       Host Port  --                     
      CTE0.A5.P2  Normal         Link Down       Host Port  --                     
      CTE0.A5.P3  Normal         Link Down     Host Port  --                     
      CTE0.B5.P0  Normal         Link Up         Host Port  8000                   
      CTE0.B5.P1  Normal         Link Down       Host Port  --                     
      CTE0.B5.P2  Normal         Link Down       Host Port  --                     
      CTE0.B5.P3  Normal         Link Down       Host Port  --                   
      WWN               Role         Working Mode  Configured Mode  Enabled  Max Speed(Mbps) 
      ----------------  -----------  ------------  ---------------  -------  --------------- 
      2408112233xxxxxx  INI and TGT  FC-AL         Auto-Adapt       Yes      8000            
      2408112233xxxxxx  INI and TGT  --            Auto-Adapt       Yes      8000            
      240a112233xxxxxx  INI and TGT  --            Auto-Adapt       Yes      8000            
      240b112233xxxxxx  INI and TGT  --            Auto-Adapt       Yes      8000            
      2418112233xxxxxx  INI and TGT  FC-AL         Auto-Adapt       Yes      8000            
      2419112233xxxxxx  INI and TGT  --            Auto-Adapt       Yes      8000            
      241a112233xxxxxx  INI and TGT  --            Auto-Adapt       Yes      8000            
      241b112233xxxxxx  INI and TGT  --            Auto-Adapt       Yes      8000  

After checking the aforementioned specifications related to front-end ports, proceed to the following aspects:

  • Check that the number of connected front-end ports on one controller must be the equal to that on the other controller, and the connected ports must be evenly selected from multiple front-end interface modules for load balancing between controllers and between front-end interface modules. For example, controller A is equipped with two 8 Gbit/s Fibre Channel interface modules, and a switch is connected to controller A through four optical fibers. In this case, each Fibre Channel interface module is connected to two optical fibers.
  • Check that the working rates of front-end ports displayed on OceanStor DeviceManager or on the CLI are consistent with the actual specifications. If degradation occurs, troubleshoot the network, switches, and port modules and replace the faulty component.
  • Check that the displayed work mode complies with the actual connection mode. P2P indicates a switch-based connection mode, and FC-AL indicates a direction connection mode.

Checking the Concurrency Pressure of Front-End Ports

To reach optimal performance, ensure that the host side provides a sufficient concurrency pressure. If the number of concurrent tasks on the host side is proportionately large whereas the performance (IOPS and/or bandwidth) is not high, it is possible that the host pressure is not transferred to the front end of the storage system or the storage system has reached a bottleneck.

To check the front-end concurrency pressure of the storage system, use either of the methods below:

  • Method 1: Use a formula.

    This method is suitable for scenarios where the pressure is fixed. For example, use IOmeter or Vdbench to test performance under a fixed pressure. During the test, the number of concurrent tasks is typically fixed. After observing the IOPS and latency, use the formula of IOPS = Number of concurrent tasks x 1000/latency (ms) to obtain the number of concurrent tasks to find out the front-end concurrency pressure. For example, if the IOPS is 3546 and the latency is 6.77 ms, the number of concurrent tasks is 24 (namely, 3546 x 6.77/1000).

  • Method 2: Run the CLI command to obtain an approximate number of front-end concurrent tasks.

    This method is suitable for scenarios that involve a changing pressure. The show controller io io_type=frontEnd controller_id=XX command is used to query the front-end concurrent I/O tasks delivered to the specified controller. Run this command multiple times and use a stable value as an approximate number of front-end concurrent tasks. XX indicates a controller ID.

     
    admin:/>show controller io io_type=frontEnd controller_id=0A  
     
      Controller Id : 0A      
      Front End IO    : 0       
      Front End Limit : 17408 
NOTE:

If the latency is low, the number of front-end concurrent tasks obtained by running show controller io may be inaccurate. In this case, use method 1 instead.

If the concurrency pressure is insufficient at the front end, this can be increased on the host side. If the pressure remains to be insufficient after this, locate the problem on the host side.

Checking Bit Errors of the Front-End Ports

If the performance fluctuates frequently or drops dramatically, there may be problems at the front-end ports or links.

Use the OceanStor DeviceManager, CLI, or inspection report function to check whether these ports have bit errors.

NOTE:

Error codes of front-end ports vary with different versions. The actual interface prevails.

  • Use OceanStor DeviceManager to view the bit errors of front-end ports (Fibre Channel/iSCSI).

  • Run show port bit_error to view the bit errors of front-end ports.
    admin:/>show port bit_error 
    ETH port: 
     
      ID                     Error Packets  Lost Packets  Over Flowed Packets  Start Time                     CRC Errors  Frame Errors  Frame Length Errors 
      ---------------------  -------------  ------------  -------------------  -----------------------------  ----------  ------------  ------------------- 
      CTE0.A2.P0             0              0             0                    2017-06-06/18:48:32 UTC+08:00  0           0             0 
      CTE0.A2.P1             0              0             0                    2017-06-06/18:48:31 UTC+08:00  0           0             0 
      CTE0.A2.P2             0              0             0                    2017-06-06/18:48:31 UTC+08:00  0           0             0 
      CTE0.A2.P3             0              0             0                    2017-06-06/18:48:31 UTC+08:00  0           0             0 
      CTE0.B2.P0             0              0             0                    2017-06-06/18:48:42 UTC+08:00  0           0             0 
      CTE0.B2.P1             0              0             0                    2017-06-06/18:48:41 UTC+08:00  0           0             0 
      CTE0.B2.P2             0              0             0                    2017-06-06/18:48:41 UTC+08:00  0           0             0 
      CTE0.B2.P3             0              0             0                    2017-06-06/18:48:41 UTC+08:00  0           0             0 
      CTE0.B5.P0             0              0             0                    2017-06-06/18:48:48 UTC+08:00  0           0             0 
      CTE0.B5.P1             0              0             0                    2017-06-06/18:48:48 UTC+08:00  0           0             0 
      CTE0.B5.P2             0              0             0                    2017-06-06/18:48:48 UTC+08:00  0           0             0 
      CTE0.B5.P3             0              0             0                    2017-06-06/18:48:48 UTC+08:00  0           0             0 
      CTE0.SMM0.MGMT         0              0             0                    2017-06-06/18:48:15 UTC+08:00  0           0             0 
      CTE0.SMM1.MGMT         0              0             0                    2017-06-06/18:48:20 UTC+08:00  0           0             0 
      CTE0.SMM0.MAINTENANCE  0              0             0                    2017-06-06/18:48:17 UTC+08:00  0           0             0 
      CTE0.SMM1.MAINTENANCE  0              0             0                    2017-06-06/18:48:20 UTC+08:00  0           0             0 
    FC port: 
     
      ID          Lost Signals  Link Errors Codes  Lost Synchronizations  Failed Connections  Start Time                     CRC Errors  Frame End Sign Errors 
      ----------  ------------  -----------------  ---------------------  ------------------  -----------------------------  ----------  --------------------- 
      CTE0.A4.P0  0             0                  0                      0                   2017-06-06/18:48:40 UTC+08:00  0           0 
      CTE0.A4.P1  0             0                  0                      0                   2017-06-06/18:48:40 UTC+08:00  0           0 
      CTE0.A4.P2  0             0                  0                      0                   2017-06-06/18:48:40 UTC+08:00  0           0 
      CTE0.A4.P3  0             0                  0                      0                   2017-06-06/18:48:40 UTC+08:00  0           0 
      CTE0.B4.P0  0             0                  0                      0                   2017-06-06/18:48:47 UTC+08:00  0           0 
      CTE0.B4.P1  0             0                  0                      0                   2017-06-06/18:48:47 UTC+08:00  0           0 
      CTE0.B4.P2  0             0                  0                      0                   2017-06-06/18:48:47 UTC+08:00  0           0 
      CTE0.B4.P3  0             0                  0                      0                   2017-06-06/18:48:47 UTC+08:00  0           0 
      CTE0.B6.P0  0             0                  0                      0                   2017-06-06/18:48:48 UTC+08:00  0           0 
      CTE0.B6.P1  0             0                  0                      0                   2017-06-06/18:48:48 UTC+08:00  0           0 
      CTE0.B6.P2  0             0                  0                      0                   2017-06-06/18:48:48 UTC+08:00  0           0 
      CTE0.B6.P3  0             0                  0                      0                   2017-06-06/18:48:47 UTC+08:00  0           0 
    SAS port: 
     
      ID           Invalid Dword  Consist Errors  Loss Of DWORD  PHY Reset Errors  Start Time 
      -----------  -------------  --------------  -------------  ----------------  ----------------------------- 
      CTE0.A0.P0 0              0               0              0                 2017-06-06/18:48:31 UTC+08:00 
      CTE0.A0.P1 0              0               0              0                 2017-06-06/18:48:31 UTC+08:00 
      CTE0.A0.P2 0              0               0              0                 2017-06-06/18:48:31 UTC+08:00 
      CTE0.A0.P3 0              0               0              0                 2017-06-06/18:48:31 UTC+08:00 
      CTE0.B0.P0 0              0               0              0                 2017-06-06/18:48:33 UTC+08:00 
      CTE0.B0.P1 0              0               0              0                 2017-06-06/18:48:33 UTC+08:00 
      CTE0.B0.P2 0              0               0              0                 2017-06-06/18:48:33 UTC+08:00 
      CTE0.B0.P3 0              0               0              0                 2017-06-06/18:48:33 UTC+08:00 
      DAE000.A.P0  2724           2798            80             108               2017-06-06/18:49:01 UTC+08:00 
      DAE000.A.P1  2666           2733            77             116               2017-06-06/18:49:01 UTC+08:00 
      DAE000.B.P0  2537           2603            75             109               2017-06-06/18:49:00 UTC+08:00 
      DAE000.B.P1  4736           4792            60             108               2017-06-06/18:49:01 UTC+08:00 
    FCoE port: 
    PCIE port: 
     
      ID          ECRC Error  Bad Tlp  Bad Dllp  Recv Error  Chip Ecc Error  Start Time                     Replay Timer Timeout  Rollover 
      ----------  ----------  -------  --------  ----------  --------------  -----------------------------  --------------------  -------- 
      CTE0.A3.P0  0           0        0         0           0               2017-06-06/18:48:31 UTC+08:00  0                     0 
      CTE0.A3.P1  0           0        0         0           0               2017-06-06/18:48:31 UTC+08:00  0                     0 
      CTE0.B3.P0  0           0        0         0           0               2017-06-06/18:48:33 UTC+08:00  0                     0 
      CTE0.B3.P1  0           0        0         0           0               2017-06-06/18:48:33 UTC+08:00  0                     0

If the bit errors rise continuously, there is a performance problem in the front-end port or its links. In this case, replace the optical fiber or optical module.

Checking Performance Indicators of Front-End Ports

The following are performance indicators of the front-end ports: average read I/O response time, average write I/O response time, average I/O size, IOPS, and bandwidth.

o query the performance indicators of front-end ports, use OceanStor DeviceManager or SystemReporter, or run the CLI command.

NOTE:

Performance indicators of front-end ports vary with different versions. However, the actual interface prevails.

  • Use OceanStor DeviceManager to query performance indicators of front-end ports.

    Operation path: Monitor > Performance Monitoring > Front-end host port.

  • Use SystemReporter to query performance indicators of front-end ports.

    Operation path: Monitoring > Real-Time Monitoring > Front-End Host Port.

  • Run show performance port on the CLI to query performance indicators of front-end ports.
    dmin:/>show performance port port_id=CTE0.A4.P0 
    0.Max. Bandwidth(MB/s) 
    1.Usage Ratio(%) 
    2.Queue Length 
    3.Bandwidth(MB/s) / Block Bandwidth(MB/s) 
    4.Throughput(IOPS)(IO/s) 
    5.Read Bandwidth(MB/s) 
    6.Average Read I/O Size(KB) 
    7.Read Throughput(IOPS)(IO/s) 
    8.Write Bandwidth(MB/s) 
    9.Average Write I/O Size(KB) 
    10.Write Throughput(IOPS)(IO/s) 
    11.Read I/O Granularity Distribution: [0K,1K)(%) 
    12.Read I/O Granularity Distribution: [1K,2K)(%) 
    13.Read I/O Granularity Distribution: [2K,4K)(%) 
    14.Read I/O Granularity Distribution: [4K,8K)(%) 
    15.Read I/O Granularity Distribution: [8K,16K)(%) 
    16.Read I/O Granularity Distribution: [16K,32K)(%) 
    17.Read I/O Granularity Distribution: [32K,64K)(%) 
    18.Read I/O Granularity Distribution: [64K,128K)(%) 
    19.Read I/O Granularity Distribution: [128K,256K)(%) 
    20.Read I/O Granularity Distribution: [256K,512K)(%) 
    21.Read I/O Granularity Distribution: >= 512K(%) 
    22.Write I/O Granularity Distribution: [0K,1K)(%) 
    23.Write I/O Granularity Distribution: [1K,2K)(%) 
    24.Write I/O Granularity Distribution: [2K,4K)(%) 
    25.Write I/O Granularity Distribution: [4K,8K)(%) 
    26.Write I/O Granularity Distribution: [8K,16K)(%) 
    27.Write I/O Granularity Distribution: [16K,32K)(%) 
    28.Write I/O Granularity Distribution: [32K,64K)(%) 
    29.Write I/O Granularity Distribution: [64K,128K)(%) 
    30.Write I/O Granularity Distribution: [128K,256K)(%) 
    31.Write I/O Granularity Distribution: [256K,512K)(%) 
    32.Write I/O Granularity Distribution: >= 512K(%) 
    33.Average IO Size(KB) 
    34.% Read 
    35.% Write 
    36.Max IOPS(IO/s) 
    37.Service Time(Excluding Queue Time)(us) 
    38.Average I/O Latency(us) 
    39.Max. I/O Latency(us) 
    40.Average Read I/O Latency(us) 
    41.Average Write I/O Latency(us) 
    42.Sending bandwidth for replication(KB/s) 
    43.Receiving bandwidth for replication(KB/s) 
    44.The cumulative count of I/Os 
    45.The cumulative count of data transferred in Kbytes 
    46.The cumulative elapsed I/O time(ms) 
    47.The cumulative count of all reads 
    48.The cumulative count of all writes 
    Input item(s) number seperated by comma:

    Analyzing performance indicators of the front-end ports can help in the identification of associating problems.

  • Average read/write latency: When locating a performance problem, first compare the average latency of the front-end ports you have queried with that observed on the host side. Check if the values differ greatly, and determine if the performance problem lies in the storage side. In addition, compare the latency of front-end ports with that of back-end disk domains and disks to determine I/O characteristics. For example, in a read service scenario, if the latency of the front-end port is much lower than that at the back-end port, sequential read I/Os may be delivered or the hit ratio may be high. However, if the latencies at both ends are similar, random read I/Os may be delivered.
    NOTE:

    The measured latency at the front-end I/O does not include that which has been caused by the interaction between the host and the storage system and I/O transmission over links.

  • Average I/O size: Indicates the average size of I/Os received by the storage system. If this is different from that delivered by a host, then the host side or the HBA driver has split or combined I/Os.
  • IOPS and bandwidth: Determines if the service pressure is balanced at the front-end port by checking the connection ports related to IOPS. Also check that the pertaining bandwidth is close to the corresponding maximum level for links. In these two levels of bandwidth are same or similar, it creates a bottleneck in performance.
Translation
Download
Updated: 2019-07-17

Document ID: EDOC1100049152

Views: 11477

Downloads: 77

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next