No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade
Knowledge Base

Backup troughput performance issue on OceanStor 5500V3 storage

Publication Date:  2016-05-02  |   Views:  307  |   Downloads:  0  |   Author:  SU1003432122  |   Document ID:  EKB1000115628

Contents

Issue Description

Storage version:V300R001C20

Total Capacity:107TB

The customer are trying to use ArcServe UDP(Unified Data Protection) backup server restore backup data to vCenter(ESXi hosts are connected to 5500V3 storage through 8Gbps Fiber Channel), the transferring data throughput speed is only about 10MB/s. But the same system in another seperate site has about 113MB/s backup throughput speed.

As the customer's description:

Site A:

Backup server to 5500V3  => 10MB/s

Backup server to different storage/server => 113MB/s

Site B:

Backup server to 5500V3  => 113MB/s

Backup server to different storage/server => 113MB/s

Alarm Information

None

Handling Process

1. Since there're two same systems with totally different performance, also the storage has 107TB capacity, the write performance never could be 10MB/s, we presume there's configuration problem somewhere. So, we directly ask the customer for a remote session.

2. Through the remote session, we check the real-time performance statistics monitoring first. When the customer start to backup data, the IO latency on storage only 1~2ms, the total throughput is about 15MB/s. So the performance bottleneck is not storage.

3. Check the performance staitistics on VMware vCenter. The IO latency is also 1~2ms, the performance bottleneck is not VMware either.

4. Then, we ask the customer show us the backup procedure. The customer login a Windows VM first, then access the backup server through the CIFS share, finally copy data from backup server to local disk on VM(The disk space is on storage). So, we excluded the target side of the backup operation.

5. Next, we check the source side of the backup operation. The backup server is using the LUN from another 5500V3. The same as step 2 and 3, we exclude the source side.

6. Since the customer backup data from CIFS sharing network, the only possibility is the networking issue. But the customer don't agree with that, because they're using 1Gbps switch. And, from the same backup server to another server or storage has a throughput speed of 113MB/s.

So, we login the switch and check the speed of each port. Finally, we found one of the port connected to backup server is 100Mbps speed.

Root Cause

The backup server has serval ETH ports, different port with different IP address and VLAN. Unfortunately, the port which used to communicate with VMware vCenter is 100Mbps. So, we only have 10MB/s bakcup speed, the bottleneck is network.

Solution

Replace the cable and the port speed recover to 1Gbps automatically.

Suggestions

1. The backup performance issue could be caused by any part of the backup pathing, target side, soure side and backup path. We need to check them one by one, level by level.

2. There's experience that 1Gbps network has about 110MB/s throughput, 100Mbps network is about 10MB/s and 8Gbps FC is about 800MB/s. If the bottleneck is networking, we can associate the throughput with network speed first. For example, if backup bandwidth is 35MB/s, it could never be networking issue.