No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

N8300 Filesystem expansion takes too long time

Publication Date:  2014-09-30 Views:  81 Downloads:  0
Issue Description
Online capacity expansion of a file system affects the client CIFS service access, and execution of capacity expansion is hung for a long time.
Handling Process
1 Restart the the N8300 system, after that the access to all other file systems are recovered, except filesystem 01.
2 Checked system status,we found that the capacity expansion of the filesystem 01 is already complete, but it cannot provide service until finishing the fsck command.
3 Since fsck command would take a long time to be finished, created snapshot for filesystem 01 and share with clients to be access in read only mode temporarily.
4 Execute fsck -t vxfs -o full -y /dev/vx/dsk/sfsdg/fsname to start the check. After fsck command finish, file system is recovered succeeded, all shares will be accessed properly.

Root Cause
1. I/Os are frozen during the file system expansion process. The time spent by expansion depends on many factors. The internal expansion process and impact are as follows:
(1) I/Os of the file system are frozen.
I/Os of the file system are frozen to ensure data consistency during the expansion process. fsvoladm/vxresize/fsdm sends a freeze command to all nodes in the cluster. After all nodes do not have active I/Os, the file system goes to the frozen state.
The freeze process of the file system consists of 10 layers (layer 1 to layer 10).  IOs will be frozen from layer by layer. As for applications, when the freeze operation starts, the system can sense the suspension of the applications because system invocation is blocked.
One layer can be frozen only after operations at this layer are complete and exit. The freeze operation time depends on the system invocation. If a large number of invocation requests are in the kernel, the system can be frozen only after these requests are processed.
When the system is frozen, buffer and page cache of the file system are flushed, which generates a large number of I/Os. Page flushing also consumes some time and may affect the system performance within a short period.
When the capacitywas being expanded at the IMG site, the system undertakes large service pressure from clients, and large number of small I/Os are dispersed at different layers. For this reason, capacity expansion took a long time.
(2) Defragmentation is implemented to release space.
If the file system is re-sized, some metadata must be allocated to the file system. When the file system has only a little free space but many fragments, metadata allocation will be additional workload. If the free space is fragmented or only a little free space exists, the file system does not have sufficient space to store the re-sized metadata. As a result, the file system will adjust its data space to store the re-sized metadata. This operation is similar to defragmentation of a certain number of fragments. Expansion of larger capacity requires more space to store metadata. Accordingly, defragmentation will consume longer time.
After the file system is expanded, the file system moves some metadata to new space, prolonging the resizing process.
(3) The file system is remounted.
When the file system is remounted, the file system reads inode list from disks and initializes the inode list structure again. If the file system has many files, the process will take a long time. The completion time of the re-size operation will be affected.
After the file system is remounted, the re-size operation can be used to unfreeze the file system and recover read and write I/Os.
Suggestions
1. It is advised to expand a file system when the file system is idle.
According to the laboratory verification statistics, when a file system is idle, it can be expanded in a few minutes. If the file system is under large service pressure, the capacity expansion may take seven hours.
2. It is advised to expand a file system when the utilization rate of the file system is not very high.
95% is the upper limit for capacity expansion of a file system. You are advised to expand a file system when its utilization rate does not exceed 80%.
3. It is advised to expand capacity that is not so large at a time.
When the space utilization rate of a file system is high, you are advised to expand the file system for multiple times, e.g. one TB for each time. 

END