AWS Storage Blog
Optimizing storage management with Amazon FSx for Lustre storage quotas
Data-intensive use cases across industries are getting larger each year. Use cases that require multi-user file systems, such as user shares for data science or computational engineering, can benefit from cloud solutions to help manage file storage consumption.
In this blog post, I walk you through the Amazon FSx for Lustre storage quota feature, discussing quota parameters and quota administration commands. This feature enables customers to limit the amount of disk space and the number of files that a user or a group can use. It can set quotas on both users, and groups consuming several files and disk space resources.
Storage quota administration
You can use Amazon FSx for Lustre storage quotas to manage and regulate the storage usage of set of users and/or groups in your organization. The storage quotas are useful for organizations that use Amazon FSx for Lustre across multiple sets of users running own separate workloads. After mounting the FSx for Lustre file system, you can administer storage quotas using Lustre File System (LFS) quota control commands on Lustre clients. Root users can set quotas on the lustre client for individual users and groups, though it is not possible to enforce storage quotas on the root user. If the group quota exists on the file system, Amazon FSx for Lustre checks the quota of the creator’s group and the file size. Those factors then count towards the group’s quota before writing the file to the partition. If no group quota exists, the storage quota checks the user quota before writing the file to the blocks (Object Storage Targets – OSTs). The system administrator can control the over-uses of the users and groups allocated space by setting up the blocks, or inode (Metadata Targets – MDTs) quotas.
Three quota parameters control storage quota administration: soft limits, grace periods, and hard limits.
Soft limits
Soft limits are a numerical value set by the administrator to track user and group level usage. The user or group can still allocate blocks or inode even after exceeding the soft limit, but once it exceeds the grace period, the timer starts. When the grace period time expires, the user limits are checked. If it is still higher than the soft limit, the soft limit then becomes a hard limit and stops the user or group from allocating any new blocks or inodes. The only way to keep under the soft limit is to delete the files by user or group, or configure a larger soft limit that is smaller than the hard limit.
Grace periods
Storage quotas use grace periods in the process of setting up soft limits for users and groups. Grace periods enable users and groups to go beyond their assigned soft limits for a certain period of time, which is set in seconds. Grace periods can be set at the user metadata and data levels.
Hard limits
Hard limits are the absolute limit set. The administrator can modify the hard limit to a higher quota limit for users. When a user has reached the hard limit, all future writes on the block or inode prompts them with an error, “quota exceeded” (EDQUOT).
Storage quotas solution
The system administrators can use lfs commands on Amazon EC2 Lustre client systems to work with and configure storage quotas.
lfs quota: Use this command to display general quota information, including disk usage and limits for users and groups.
lfs quota [-q] [-v] [-h] [-o obd_uuid] [-u|-g|-p uname|uid|gname|gid|projid] /mount_point
lfs quota -t {-u|-g|-p} /mount_point
lfs setquota: Use this command to set the storage quota limits and fine-tune the three quota parameters (grace periods, soft limits, and hard limits).
lfs setquota {-u|--user|-g|--group|-p|--project} username|groupname [-b block-softlimit] \
[-B block_hardlimit] [-i inode_softlimit] \
[-I inode_hardlimit] /mount_point
Note: As of the publishing of this post, Amazon FSx for Lustre does not support default quotas or project quotas.
lfs Storage quota command examples
Ensure that you have created and mounted the Amazon FSx for Lustre file system to the Amazon EC2 instance, as per these instructions. You can connect to the Amazon EC2 Lustre client instance using EC2InstanceConnect.
Use the following command to display general quota information, including disk usage and limits for the user running the command and the primary group:
$ sudo lfs quota /mnt/fsx Disk quotas for usr root (uid 0): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 2419350 0 0 - 416058 0 0 - Disk quotas for grp root (gid 0): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 2419350 0 0 - 416058 0 0 -
You can execute the command to display general quota information for a specific user:
$ sudo lfs quota -u ec2-user /mnt/fsx Disk quotas for usr ec2-user (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 25* 1 2 - 1 1 2 -
You can use the following command to list the detailed quota statistics for each MDT and OST for a specific user. You can see the metadata usage on the MDTs and storage usage on OSTs.
$ sudo lfs quota -u ec2-user -v /mnt/fsx Disk quotas for usr ec2-user (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 25* 1 2 - 1 1 2 - fsx-MDT0000_UUID 25 - 0 - 1* - 1 - fsx-OST0000_UUID 0 - 0 - - - - - fsx-OST0001_UUID 0 - 0 - - - - - fsx-OST0002_UUID 0 - 0 - - - - - fsx-OST0003_UUID 0 - 0 - - - - - fsx-OST0004_UUID 0 - 0 - - - - - fsx-OST0005_UUID 0 - 0 - - - - -
You can use the following command to display general quota information for a specific group:
$ sudo lfs quota -g testgroup /mnt/fsx Disk quotas for grp testgroup (gid 1002): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 0 0 0 - 0 0 0 -
You can use the following command to display block and inode grace times for user quotas (the default block and inode grace time is one week):
$ sudo lfs quota -t -u /mnt/fsx Block grace time: 1w; Inode grace time: 1w
lfs commands for setting limits
The setquota command is used to set file system quotas for user or group levels. A system administrator can change user and group grace time setting using this command.
Setting user quotas
In this example, the quota for user “ec2-user” is set to 2044 kilobytes, and the inode hard limit is 2001 files:
$ sudo lfs setquota -u ec2-user -b 2044 -B 2048 -i 2001 -I 2001 /mnt/fsx
To retrieve the quota allocated for “ec2-user,” You can execute the following command on the Lustre client:
$ sudo lfs quota -u ec2-user -v /mnt/fsx Disk quotas for usr ec2-user (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 25 2044 2048 - 1 2001 2001 - fsx-MDT0000_UUID 25 - 0 - 1* - 1 - fsx-OST0000_UUID 0 - 0 - - - - - fsx-OST0001_UUID 0 - 0 - - - - - fsx-OST0002_UUID 0 - 0 - - - - - fsx-OST0003_UUID 0 - 0 - - - - - fsx-OST0004_UUID 0 - 0 - - - - - fsx-OST0005_UUID 0 - 0 - - - - -
Setting group quotas
In this example, the quota for group “testgroup” is set to 2044 kilobytes, and the inode hard limit is 2001 files.
$ sudo lfs setquota -g testgroup -b 2044 -B 2048 -i 2001 -I 2001 /mnt/fsx
The following command displays quota allocated by the group “testgroup.”
$ sudo lfs quota -g testgroup -v /mnt/fsx Disk quotas for grp testgroup (gid 1002): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 0 2044 2048 - 0 2001 2001 - fsx-MDT0000_UUID 0 - 0 - 0 - 0 - fsx-OST0000_UUID 0 - 0 - - - - - fsx-OST0001_UUID 0 - 0 - - - - - fsx-OST0002_UUID 0 - 0 - - - - - fsx-OST0003_UUID 0 - 0 - - - - - fsx-OST0004_UUID 0 - 0 - - - - - fsx-OST0005_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0
Changing a user’s grace time
User has to log in and configure the grace time for blocks and inodes.
Use the following command to set a grace time for all users. In this example, I use a block grace time of 1000 seconds and an inode grace time of 0 weeks 4 days.
$ sudo lfs setquota -t -u -b 1000 -i 0w4d /mnt/fsx
To validate and view the quota that was just set for all users, run the following quota command to report the quotas usage:
$ sudo lfs quota -t -u /mnt/fsx lock grace time: 16m40s; Inode grace time: 4d
Changing a group’s grace time
Use the following command to set a grace time for all groups. In this example, I use a block grace time of 1000 seconds and an inode grace time of 0 weeks 4 days.
$ sudo lfs setquota -t -g -b 1000 -i 0w4d /mnt/fsx
To validate and view the quota that was just set for all groups, run the quota command to report the quotas usage:
$ sudo lfs quota -t -g /mnt/fsx Block grace time: 16m40s; Inode grace time: 4d
Testing the quota limits and grace periods
As a reminder, the storage quota is in place, with 2044 kilobytes for block and 2001 files for inode for the user “ec2-user,” and the grace period is set it for 1 week.
To validate and view the quota that was just set for a particular user “ec2-user,” run the following quota command to report the usage of the quota:
$ sudo lfs quota -u ec2-user -v /mnt/fsx Disk quotas for usr ec2-user (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 25 2044 2048 - 1 2001 2001 - fsx-MDT0000_UUID 25 - 0 - 1 - 1024 - fsx-OST0000_UUID 0 - 0 - - - - - fsx-OST0001_UUID 0 - 0 - - - - - fsx-OST0002_UUID 0 - 0 - - - - - fsx-OST0003_UUID 0 - 0 - - - - - fsx-OST0004_UUID 0 - 0 - - - - - fsx-OST0005_UUID 0 - 0 - - - - - Total allocated inode limit: 1024, total allocated block limit: 0
You can execute the following command to display the block grace time:
$ sudo lfs quota -t -u /mnt/fsx Block grace time: 1w; Inode grace time: 1w
You can execute the following script to create a temporary “test_storage_quota_file” of 2 MB to test the storage quota limit:
$ cd /mnt/fsx $ dd if=/dev/zero of=test_storage_quota_file bs=2 count=100000
You can check the storage quota for the “ec2-user,” and verify the grace period timer once after the creation of temporary “test_storage_file”:
$ sudo lfs quota -u ec2-user -v /mnt/fsx Disk quotas for usr ec2-user (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /mnt/fsx 1973* 2044 2048 6d23h25m21s 13 2001 2001 - fsx-MDT0000_UUID 31 - 0 - 13 - 1024 - fsx-OST0000_UUID 25* - 25 - - - - - fsx-OST0001_UUID 25* - 25 - - - - - fsx-OST0002_UUID 1821 - 2048 - - - - - fsx-OST0003_UUID 25* - 25 - - - - - fsx-OST0004_UUID 25* - 25 - - - - - fsx-OST0005_UUID 21* - 21 - - - - - Total allocated inode limit: 1024, total allocated block limit: 2169
The * shows that the user has exceeded the quota limit.
Testing block and inode quota limit
The following is an example for creating 10,000 files of 10 bytes each from “test_storage_quota_file,” which had a limit of 2 MB. The use case is to test the inode quota limit of 2001 files and 2044 kilobytes for the block by splitting the temporary file “test_storage_quota_file.” It creates random files starting with “xaaaaaa.” When the quota limit exceeds, it prompts with an error “Disk quota exceeded,” and creates files with 0 bytes in the Amazon FSx for Lustre mounted file system.
$ split -b 10 -a 10 test_storage_quota_file split: xaaaaaaaaaa: Disk quota exceeded
The total allocated block limit is the sum of all the block limits allocated to OSTs, and the inode quota per MDTs. When a user or group has fully consumed their storage quota, from one OST or MDT, the user or group may not be able to create new files. Their ability to create new files is lost regardless of the quota available on other OSTs or MDTs.
Amazon FSx for Lustre file systems allocates storage quota space in fixed sizes called qunits, the minimum qunit size is 1-M bytes for block space and 1-K inodes for inode space. It is not possible to set the quota limit below the minimal qunit size.
The following error appears when trying to set the quota limit below the minimal qunit size:
$ sudo lfs setquota -u ec2-user -b 1024 -B 1028 -i 1 -I 2 /mnt/fsx warning: block softlimit is smaller than the miminal qunit size, please see the help of setquota or Lustre manual for details. warning: inode softlimit is smaller than the miminal qunit size, please see the help of setquota or Lustre manual for details. warning: inode hardlimit is smaller than the miminal qunit size, please see the help of setquota or Lustre manual for details.
To avoid overruns of quota size, it is advisable to check the workloads of users and groups before designing the storage quota limits and grace periods.
Performance impact of using storage quotas on your file systems
You may be wondering about the potential performance impact of using storage quotas with Amazon FSx for Lustre. Even when a storage quota is set for a user or group, or the quotas are close to the limits, there are no noticeable performance impacts identified on an Amazon FSx file system.
Cleaning up
Once the storage management test using storage quotas is finished, be sure to remove the deployed resources that facilitated the test. Delete the Amazon EC2 instance and Amazon FSx for Lustre file system in the account to avoid incurring future costs on EC2 and Amazon FSx. Unless you are going to reuse the resources later.
To clean up your Amazon FSx for Lustre file system, you can run the following command in the AWS CloudShell console:
$ aws fsx delete-file-system \ --file-system-id <<insert your file system id>>
To clean up your Amazon EC2 instance, you can run the following command in the AWS CloudShell console:
$ aws ec2 terminate-instances \ --instance-ids <<insert your ec2 instance id>>
Conclusion
In this blog, you learned how to use Storage quotas to monitor and control user and group level storage consumption on the lustre file systems. You created an Amazon FSx for Lustre file system as part of the testing and validating storage quotas. Then, you used lfs commands to set the quota limits and validated by listing the quota values. You can use Storage quotas for managing multiuser file systems such as user shares for data researchers, data scientists, computational engineers, and genomics researchers.
You can now use Amazon FSx for Lustre storage quotas on all Amazon FSx for Lustre file systems, and are available in all public AWS Regions.
If you would like to read more, check out the Amazon FSx for Lustre product page, user guide, and pricing page.
Thanks for reading this blog post on storage quotas for Amazon FSx for Lustre file systems. If you have any comments or questions, please do not hesitate to leave them in the comments section.