No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

Slow write on NFS share on OceanStor 5500V3

Publication Date:  2018-01-30 Views:  79 Downloads:  0
Issue Description

The customer configured an a NFS v3 share, read-write, Synchronous, no_all_squash, root_squash and insecure port verification on an Oceanstor 2600 V3, version V300R006C00, patch SPC100 and serial 2102350SHS4YH7900025.

It was planned that this share to be used as web server repository, and when reading some file to be fast (0.002s)

However, writing tasks are very slow. On linux client we did “touch prod2.php” revealing too much time spent on LOOKUP, CREATE and SETATTR tasks:

12:13:11.582065 IP 10.33.2.253.1018 > 10.33.3.8.2049: Flags [P.], seq 796:964, ack 1157, win 716, length 168: NFS request xid 4187189069 164 lookup fh Unknown/010009000300000000000000010000000000000034D6582E6A44002101000000 "prod2.php"

12:13:11.582575 IP 10.33.3.8.2049 > 10.33.2.253.1018: Flags [P.], seq 1157:1277, ack 964, win 5594, length 120: NFS reply xid 4187189069 reply ok 116 lookup ERROR: No such file or directory

12:13:11.582587 IP 10.33.2.253.1018 > 10.33.3.8.2049: Flags [.], ack 1277, win 716, length 0

12:13:11.582624 IP 10.33.2.253.1018 > 10.33.3.8.2049: Flags [P.], seq 964:1164, ack 1277, win 716, length 200: NFS request xid 4203966285 196 create fh Unknown/010009000300000000000000010000000000000034D6582E6A44002101000000 "prod2.php"

12:13:11.621488 IP 10.33.3.8.2049 > 10.33.2.253.1018: Flags [.], ack 1164, win 5594, length 0

12:13:11.829641 IP 10.33.3.8.2049 > 10.33.2.253.1018: Flags [P.], seq 1277:1573, ack 1164, win 5594, length 296: NFS reply xid 4203966285 reply ok 292 create fh Unknown/010009000300000000000000010000000000000034D6582E6A44002101000000

12:13:11.829803 IP 10.33.2.253.1018 > 10.33.3.8.2049: Flags [P.], seq 1164:1344, ack 1573, win 716, length 180: NFS request xid 4220743501 176 setattr fh Unknown/010009000300000000000000010000000000000034D6582E6A44002101000000

12:13:11.830069 IP 10.33.3.8.2049 > 10.33.2.253.1018: Flags [.], ack 1344, win 5594, length 0

12:13:11.936204 IP 10.33.3.8.2049 > 10.33.2.253.1018: Flags [P.], seq 1573:1721, ack 1344, win 5594, length 148: NFS reply xid 4220743501 reply ok 144 setattr

12:13:11.975395 IP 10.33.2.253.1018 > 10.33.3.8.2049: Flags [.], ack 1721, win 716, length 0

Handling Process
We collected the logs and checked the configuration which seems to be correct. 
We tested if the slow write issues happens also on CIFS share. Made a copy test and noticed that the performance was normal. The slow write issues was manifesting only on NFS shares
We discarded multipathing load balancing misconfiguration issues
We asked the customer to run the below commands so we can see the NFS statistics:

change user_mode current_mode user_mode=developer

clear nfs nfsv3_stat

show nfs nfsv3_stat


We noticed that all write operations are very slow 

We confirmed that the write mode of the FS was write through

show space transaction file_system_id=XX controller_id=XX (replace XX with the file system id you made the test on and controller should be 0A)


A cache write policy can be classified into write-back and write-through.

  • Write-back: A write success message is returned to the host as soon as each write I/O arrives at the cache. Then, the cache sorts and combines data and write data to disks.
  • Write-through: Each write request is directly sent to its destination disk. The write performance is subject to the disk performance, especially the key disk parameters such as the disk type, rotational speed, and seek time. The reliability of write-through is higher than that of write-back. 
Root Cause

We analyzed again the logs and noticed that FS size was very small and the capacity was almost reaching the end. As the amount of available space used up, the FS change automatically the write mode to "Write-through"



 

Solution
We suggested the expansion of the FS to at least 15 Gb so that the FS can restore automatically the write mode to "Write-back"

Suggestions

When the file system fills up and/or you have below 10GB of free space, the write mode can change to write through (some additional space is required by write back mode)

END