Amazon FSx for Lustre

Amazon FSx for Lustre makes it easy and cost effective to launch and run the world’s most popular high-performance file system. Use it for workloads where speed matters, such as machine learning, high performance computing (HPC), video processing, and financial modeling.

The open source Lustre file system is designed for applications that require fast storage – where you want your storage to keep up with your compute. Lustre was built to quickly and cost effectively process the fastest-growing data sets in the world, and it’s the most widely used file system for the 500 fastest computers in the world. It provides sub-millisecond latencies, up to hundreds of gigabytes per second of throughput, and millions of IOPS.

Now as a fully managed service, Amazon FSx enables you to use Lustre file systems for any workload where storage speed matters. It eliminates the traditional complexity of setting up and managing Lustre file systems, allowing you to spin up a high-performance file system in minutes. It also provides multiple deployment options to optimize cost.

FSx for Lustre integrates with Amazon S3, making it easy to process data sets with the Lustre file system. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write changed data back to S3.

Overview: Amazon FSx for Lustre

Benefits

High performance and scalability

Amazon FSx for Lustre delivers the performance to satisfy a wide variety of high-performance workloads. The Lustre file system is optimized for data processing, with sub-millisecond latencies and throughput that scales to hundreds of gigabytes per second.

Seamless access to S3 data

Amazon FSx for Lustre integrates natively with Amazon S3, making it easy to process your cloud data sets in S3 with a high-performance POSIX interface. When linked to an S3 bucket, FSx for Lustre transparently presents S3 objects as files. FSx for Lustre tracks changes and enables you to write changed and new data on the file system back to your S3 bucket at any time.

Simplicity and full management

Amazon FSx for Lustre makes it easy to launch and run the high-performance Lustre file system in the cloud. You no longer need to worry about hardware provisioning and maintenance, software configuration, backup management, and complex performance tuning of file systems. In minutes, you can create and launch an Amazon FSx for Lustre file system by using the AWS Management Console, the AWS CLI, or an AWS SDK.

Access from Linux applications running anywhere

Amazon FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make any changes. FSx for Lustre provides a native file system interface and works as any file system does with your Linux operating system. It also provides read-after-write consistency and supports file locking. You can control access to your FSx for Lustre file systems with POSIX permissions and Amazon Virtual Private Cloud (VPC) rules. You can access your file systems from Amazon EC2 instances, and from on-premises computers using AWS Direct Connect or AWS VPN.

Multiple deployment options

Amazon FSx for Lustre offers a choice between scratch and persistent file systems for short-term and longer-term data processing. Scratch file systems are ideal for temporary storage and shorter-term processing of data. Data is not replicated and does not persist if a file server fails. Persistent file systems are ideal for longer-term storage and workloads. In persistent file systems, data is replicated, and file servers are replaced if they fail.

Secure and compliant

Amazon FSx for Lustre automatically encrypts your data at-rest and in-transit. If you are subject to regulatory compliance, FSx for Lustre is PCI-DSS, ISO, and SOC compliant, and is HIPAA eligible. You can also control network access via Amazon Virtual Private Cloud (VPC) Security Group rules.

Use cases

Machine learning

Machine learning workloads use massive amounts of training data. These workloads often use shared file storage because multiple compute instances need to process the training datasets concurrently. FSx for Lustre is optimal for machine learning workloads, because it provides shared file storage with high throughput and consistent, low latencies to process the ML training datasets. FSx for Lustre is also integrated with Amazon SageMaker, allowing you to accelerate your training jobs.

High performance computing

High performance computing (HPC) enables scientists and engineers to solve complex, compute-intensive problems. HPC workloads, like oil & gas discovery, and genome analysis, process massive amounts of data that need to be accessed by multiple compute instances with high levels of throughput. FSx for Lustre is ideal for HPC and scientific computing workloads because it provides a file system that’s optimized for the performance and costs of high-performance workloads, with file system access across thousands of EC2 instances. FSx for Lustre also integrates with AWS ParallelCluster and AWS Batch, making it easy to use with your HPC workloads.

Media processing and transcoding

Media data processing workflows, like video rendering, visual effects, and media production, need the compute and storage resources to handle the massive amounts of data being created. FSx for Lustre provides the high performance and low latencies needed for processing, distributing, and analyzing digital media files.

Autonomous Vehicles

Customers developing autonomous vehicle systems often test models by running simulations and training on massive amounts of vehicle sensor and camera data to ensure vehicle safety. FSx for Lustre enables you to access that data simultaneously from thousands of nodes with high levels of performance, allowing you to more easily run simulations at scale and to accelerate model development.

Big data and financial analytics

Big data analytics use cases, including fraud detection and financial analysis, produce massive volumes of data that need high-performance storage to power data-intensive applications. Processing the ever-increasing volume of data can be costly and complicated. Amazon FSx for Lustre can process large amounts of data at scale and is performance and cost-optimized, providing faster time to discovery and value to your organization.

Electronic Design Automation (EDA)

EDA is a high-performance application used to simulate performance and failures during the design phase of silicon chip production. FSx for Lustre provides the performance and flexibility that enables you to innovate faster, design and verify new products, and scale to meet demand.

Product-Page_Standard-Icons_01_Product-Features_SqInk
Check out the Amazon FSx for Lustre features

Learn about the key features of Amazon FSx for Lustre.

Learn more 
Product-Page_Standard-Icons_02_Sign-Up_SqInk
Sign up for a free account

Instantly get access to the AWS Free Tier. 

Sign up 
Product-Page_Standard-Icons_03_Start-Building_SqInk
Start building in the console

Get started building with Amazon FSx for Lustre in the AWS Console.

Sign in