Amazon FSx for Lustre

Fully managed file system for compute-intensive workloads

Amazon FSx for Lustre is a fully managed file system that is optimized for compute-intensive workloads, such as high performance computing, machine learning, and media data processing workflows. Many of these applications require the high-performance and low latencies of scale-out, parallel file systems. Operating these file systems typically requires specialized expertise and administrative overhead, requiring you to provision storage servers and tune complex performance parameters. With Amazon FSx, you can launch and run a Lustre file system that can process massive data sets at up to hundreds of gigabytes per second of throughput, millions of IOPS, and sub-millisecond latencies.

Amazon FSx for Lustre is seamlessly integrated with Amazon S3, making it easy to link your long-term data sets with your high performance file systems to run compute-intensive workloads. You can automatically copy data from S3 to FSx for Lustre, run your workloads, and then write results back to S3. FSx for Lustre also enables you to burst your compute-intensive workloads from on-premises to AWS by allowing you to access your FSx file system over Amazon Direct Connect or VPN. FSx for Lustre helps you cost-optimize your storage for compute-intensive workloads: It provides cheap and performant non-replicated storage for processing data, with your long-term data stored durably in Amazon S3 or other low-cost data stores. With Amazon FSx, you pay for only the resources you use. There are no minimum commitments, upfront hardware or software costs, or additional fees.

Amazon FSx for Lustre Overview

Benefits

Massively scalable performance

Amazon FSx delivers the performance to satisfy your most demanding compute-intensive workloads. Amazon FSx provides throughput of up to hundreds of gigabytes per second and millions of IOPS. Amazon FSx for Lustre supports the high bandwidth needed to scale out to process petabytes of data and perform file operations with sub-millisecond latencies.

Seamless access to data on S3 or on-premises

Amazon FSx is integrated with Amazon S3, making it easy to access your S3 data to run compute-intensive workloads. Your S3 data is automatically copied to FSx for Lustre, and you can write your results back to S3. FSx for Lustre also enables you to burst your compute-intensive workloads from on-premises to AWS by allowing you to access your FSx file system over Amazon Direct Connect or VPN. This lets you run compute-intensive workloads on FSx for Lustre and store your long-term data on S3 or on-premises data stores. 

Fully managed

Amazon FSx is fully managed, making it easy to launch and run high-performance file systems in the cloud. You no longer need to worry about hardware provisioning and maintenance, software configuration, and complex performance tuning of your Lustre file system. In minutes, you can create and launch a Amazon FSx file system by using the AWS Management Console, the AWS CLI, or an AWS SDK.

Cost-effective

Amazon FSx for Lustre helps you cost-optimize your storage for compute-intensive workloads: It provides cheap and performant non-replicated storage for processing data, with your long-term data stored durably in Amazon S3 or other low-cost data stores. You pay only for the resources you use. There are no minimum commitments or upfront fees.

Native file system interface

Amazon FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make any changes. FSx for Lustre provides a native file system interface and works as any file system does with your Linux operating system. It also provides read-after-write consistency and supports file locking.

Secure and compliant

Amazon FSx automatically encrypts your data-at-rest. If you are subject to regulatory compliance, FSx for Lustre is PCI-DSS and ISO compliant and HIPAA eligible. Amazon FSx supports Amazon Virtual Private Cloud (VPC), so you can launch your FSx for Lustre file system resources in your virtual network that you share between your organization's data center and AWS or with a partner organization.

How it works

FSx_Lustre_diagram

Use cases

Machine learning

Machine learning workloads use massive amounts of training data. These workloads often use shared file storage because multiple compute instances need to process the training datasets concurrently. Amazon FSx is optimal for machine learning workloads, because it provides massively parallel shared file storage with high throughput and consistent, low latencies to process the ML training datasets.

High performance computing (HPC)

High performance computing (HPC) enables scientists and engineers to solve complex, compute-intensive problems. HPC workloads, like oil & gas discovery and genomics, process massive amounts of data that need to be accessed by multiple compute instances with high levels of throughput. Amazon FSx is ideal for HPC workloads because it provides a file system that’s optimized for the performance and costs of short-lived, compute-intensive workloads, with file system access across thousands of EC2 instances.

Media processing and transcoding

Media data processing workflows, like video rendering, visual effects, and media production, need the compute and storage resources to handle the massive amounts of data being created. Amazon FSx provides the high performance and low latencies needed for processing, distributing, and analyzing digital media files.

Cloud bursting from on-premises

When running compute intensive workloads on-premises, you often need more compute and storage capacity to handle peaks in demand. Your on-premises resources may quickly become insufficient. Amazon FSx enables you to burst your on-premises compute-intensive workloads to AWS by importing data over Amazon Direct Connect and VPN.

Big data analytics

Big data analytics use cases, including seismic processing, fraud detection, genomics, and climate modeling, produce massive volumes of data that need high-performance storage to power data-intensive applications. Managing the escalating volume of data can be costly and complex. Amazon FSx for Lustre is performance and cost-optimized, resulting in faster time to discovery and value to your organization.

Electronic Design Automation (EDA)

EDA is a common HPC application used to simulate performance and failures during the design phase of silicon chip production. These compute-intensive workloads can lead to either over-provisioning and high costs or under-provisioning and capacity restrictions. FSx for Lustre provides the performance and flexibility that enables you to innovate faster, design and verify new products, and scale to meet demand.

Product-Page_Standard-Icons_01_Product-Features_SqInk
Check out the Amazon FSx for Lustre features

Learn about the key features of Amazon FSx for Lustre.

Learn more 
Product-Page_Standard-Icons_02_Sign-Up_SqInk
Sign up for a free account

Instantly get access to the AWS Free Tier. 

Sign up 
Product-Page_Standard-Icons_03_Start-Building_SqInk
Start building in the console

Get started building with Amazon FSx for Lustre in the AWS Console.

Sign in