Amazon FSx for Lustre Documentation

Amazon FSx for Lustre is a fully managed service that is designed to provide cost-effective, high-performance, scalable storage for compute workloads. FSx for Lustre is designed to offer shared storage with sub-ms latencies, up to terabytes per second of throughput, and millions of IOPS. FSx for Lustre file systems can also be linked to Amazon Simple Storage Service (Amazon S3) buckets, allowing you to access and process data concurrently from both a file system and from the Amazon S3 API.

Amazon FSx for Lustre file systems are designed to scale to terabytes per second of throughput and millions of IOPS. FSx for Lustre is also designed to support concurrent access to the same file or directory from thousands of compute instances, and to provide low latencies for file operations.

FSx for Lustre is designed to be compatible with many Linux-based AMIs, including Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux.

Amazon FSx integrates natively with Amazon S3, enabling you to easily access your S3 data to run data processing workloads.

Amazon FSx for Lustre is designed to be accessible from workloads running on Amazon EC2 instances or on on-premises computers/servers, as well as from containers running on Amazon Elastic Kubernetes Service (EKS).

Using the Amazon FSx console, CLI, or API, you can create and scale a high-performance Lustre-based file system. Amazon FSx file systems are designed to manage time-consuming administrative tasks such as managing file servers and storage volumes, updating hardware, configuring software, managing capacity, and tuning performance.

FSx for Lustre offers a choice between scratch and persistent file systems for short-term and longer-term data processing. Scratch file systems can be a good choice for temporary storage and shorter-term processing of data. Scratch file systems are designed not to replicate data and not to persist if a file server fails. Persistent file systems can be a good choice for longer-term storage and workloads. Persistent file systems are designed to replicate data and replace file servers if they fail.

FSx for Lustre offers Solid-State Disk (SSD) and Hard Disk Drive (HDD) storage options to help you optimize cost and performance for your workload. If you are selecting an HDD-based file system, you can choose to provision an SSD cache that is designed to provide sub-millisecond latencies and higher IOPS for frequently accessed files.

You can use storage quotas to monitor and control user-and group-level storage consumption on your file systems, and to help ensure that users or groups are not able to consume excessive amounts of capacity.

You can use data compression to help reduce storage consumption of both your file system storage and your file system backups. The data compression feature uses the LZ4 compression algorithm, which is designed to deliver high levels of compression without adversely impacting file system performance. Once data compression is enabled, FSx for Lustre is designed to automatically compress newly written files before they are written to disk and automatically uncompress them when they are read.

You can release inactive data from your file system after the files are exported to Amazon S3. When a file is released, the file data is removed from the file system and the metadata remains on your file system. FSx for Lustre is designed to automatically load the data back onto your file system from your S3 bucket if a user or application accesses the released file.

Amazon FSx for Lustre file systems are designed to be encrypted at-rest, and in-transit encryption is available in select regions.

You can access your Amazon FSx file system from endpoints in your Amazon VPC, which enables you to isolate your file system in your own virtual network. You can configure security group rules and control network access to your Amazon FSx file systems.

Amazon FSx is integrated with AWS Identity and Access Management (IAM). This integration means that you can control the actions your AWS IAM users and groups can take to manage your file systems (such as creating and deleting file systems). You can also tag your Amazon FSx resources and control the actions that your IAM users and groups can take based on those tags.

Amazon FSx is integrated with AWS Backup, enabling you to use fully managed, policy-based backup and restore capabilities for your Amazon FSx file systems.

You can copy your Amazon FSx file system backups across AWS Regions, AWS accounts, or both.

For additional information about service controls, security features and functionalities, including, as applicable, information about storing, retrieving, modifying, restricting, and deleting data, please see https://docs.aws.amazon.com/index.html. This additional information does not form part of the Documentation for purposes of the AWS Customer Agreement available at http://aws.amazon.com/agreement, or other agreement between you and AWS governing your use of AWS’s services.