Amazon FSx for Lustre makes it easy to launch and operate Lustre file systems that allow your workloads to process data with consistent sub-millisecond latencies, up to hundreds of gigabytes per second of throughput, and up to millions of IOPS.
FSx for Lustre offers a choice between scratch and persistent file systems for short-term and longer-term data processing. Scratch file systems are ideal for temporary storage and shorter-term processing of data. Data is not replicated and does not persist if a file server fails. Persistent file systems are ideal for longer-term storage and workloads. In persistent file systems, data is replicated, and file servers are replaced if they fail.
FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make any changes. FSx for Lustre provides a native file system interface and works as any file system does with your Linux operating system. It also provides read-after-write consistency and supports file locking.
FSx for Lustre can also be used to process cloud data sets stored in Amazon S3 by linking an S3 bucket to the file system. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write results back to S3.
FSx for Lustre also enables you to burst your data processing workloads from on-premises to AWS by allowing you to access your FSx file system over Amazon Direct Connect or VPN.
Most popular high-performance file system
The Lustre open source file system was built to solve the problem of quickly and cheaply processing the world’s ever-growing data sets, and it’s the most widely used file system for the 500 fastest computers in the world. It is battle-tested across a broad set of industries, from oil & gas to life sciences to media & entertainment to financial services, for workloads ranging from genome sequencing to video transcoding to machine learning to fraud detection.
Simple and fully managed
With a few clicks in the Amazon FSx console, CLI, or API you can create and launch a high-performance Lustre file system. With Amazon FSx file systems, you don't have to worry about managing file servers and storage volumes, updating hardware, configuring software, or tuning performance -- Amazon FSx automates these time-consuming administration tasks.
For further data protection and to meet business and regulatory compliance requirements, Amazon FSx can also automatically take incremental backups of your file system, storing the backups on Amazon S3 with 99.999999999% (11 9's) of durability.
Multiple deployment options
Amazon FSx for Lustre offers a choice between scratch and persistent file systems for short-term and longer-term data processing. Scratch file systems are ideal for temporary storage and shorter-term processing of data. Data is not replicated and does not persist if a file server fails. Persistent file systems are ideal for longer-term storage and workloads. In persistent file systems, data is replicated, and file servers are replaced if they fail.
Data accessible to other AWS services
Amazon FSx for Lustre integrates with Amazon Sagemaker as an input data source. When using Amazon SageMaker with Amazon FSx for Lustre, your machine learning training jobs are accelerated by eliminating the initial download step from S3, and your TCO is reduced by avoiding the repeated download of common objects (saving S3 request costs) for iterative jobs on the same data set.
Amazon FSx for Lustre integrates with AWS Batch though EC2 Launch Templates. AWS Batch is a cloud-native batch scheduler for HPC, ML, and other asynchronous workloads. AWS Batch will automatically and dynamically size instances to job resource requirements, and use existing FSx for Lustre file systems when launching instances and running jobs.
FSx for Lustre also integrates with AWS ParallelCluster. AWS ParallelCluster is an AWS-supported open-source cluster management tool used to deploy and manage High Performance Computing (HPC) clusters. It can automatically create FSx for Lustre file systems or use existing file systems during the cluster creation process.
Seamless integration with your Amazon S3 data
Amazon FSx integrates natively with Amazon S3, making it easy to access your S3 data to run data processing workloads.
With a few clicks in the console, you can create a file system that’s linked to an S3 bucket. After you link your S3 bucket to your file system, FSx for Lustre transparently presents S3 objects as files and allows you to write results back to S3. Amazon FSx for Lustre automatically tracks changes on your file system and enables you to write changed and new data on the file system back to your S3 bucket at any time. Each time you export your file system data, Amazon FSx only copies incremental data and its associated symbolic links, file ownership metadata, and file timestamps to S3. Amazon FSx for Lustre uses parallel data-transfer techniques to export data back to S3, enabling fast data transfer.
Amazon FSx for Lustre is accessible from compute workloads running on Amazon EC2 instances and containers running on Amazon Elastic Kubernetes Service (EKS). You access your file system from your EC2 compute instances using the open-source Lustre client. Once mounted, you can work with the files and directories in your file system just like you would with a local file system. You access FSx for Lustre from containers running on EKS using the open-source FSx for Lustre CSI driver. Your containers running on AWS EKS can use high-performance persistent volumes (PVs) backed by FSx for Lustre.
FSx for Lustre is compatible with the most popular Linux-based AMIs, including Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux. The Lustre client is included with Amazon Linux 2 and Amazon Linux. For RHEL, CentOS, and Ubuntu an AWS repository for the Lustre client is supported that provides clients compatible with these operating systems.
With FSx for Lustre, you can mix and match the instance types and Linux AMIs that are connected to a single file system.
Accessible from on-premises
You can use Amazon FSx for Lustre for on-premises compute workloads that need to burst to the cloud due to peak demands or capacity limits. To move your existing on-premises data into Amazon FSx, you can mount your Amazon FSx for Lustre file system from an on-premises client over AWS Direct Connect or VPN and then copy your data set to your Amazon FSx for Lustre file system. At any time, you can write your results back to your durable data lake storage.
Amazon FSx for Lustre provides a cost-effective and performant file system for processing data. You pay only for the resources you use, with no minimum commitments or upfront fees. You are billed in per-second increments based on the storage capacity you provision, the replication level of the file system, and the throughput level you choose to match your workloads' requirements. Amazon FSx for Lustre reduces the storage costs for you to run processing-heavy workloads by enabling you to spin-up and spin-down file systems as needed, store long-term data in Amazon S3, and automatically loading only a subset of your data that is needed for processing from a linked S3 bucket. For more details, see the Amazon FSx pricing page.
Secure and compliant
All Amazon FSx for Lustre file systems are encrypted at-rest, and in-transit encryption is available in select regions.
AWS has the longest-running compliance program in the cloud and is committed to helping customers navigate their requirements. Amazon FSx has been assessed to meet global and industry security standards. It complies with PCI DSS, ISO 9001, 27001, 27017, and 27018), and SOC 1, 2, and 3, in addition to being HIPAA eligible. For more information and resources, visit our compliance pages. You can also go to the Services in Scope by Compliance Program page to see a full list of services and certifications.
You access your Amazon FSx file system from endpoints in your Amazon VPC, which enables you to isolate your file system in your own virtual network. You can configure security group rules and control network access to your Amazon FSx file systems.
Amazon FSx is integrated with AWS Identity and Access Management (IAM). This integration means that you can control the actions your AWS IAM users and groups can take to manage your file systems (such as creating and deleting file systems). You can also tag your Amazon FSx resources and control the actions that your IAM users and groups can take based on those tags.