Amazon FSx for Lustre
High-performance file system for processing Amazon S3 or on-premises data
Amazon FSx for Lustre provides a high-performance file system optimized for fast processing of workloads such as machine learning, high performance computing (HPC), video processing, financial modeling, and electronic design automation (EDA). These workloads commonly require data to be presented via a fast and scalable file system interface, and typically have data sets stored on long-term data stores like Amazon S3.
Operating high-performance file systems typically requires specialized expertise and administrative overhead, requiring you to provision storage servers and tune complex performance parameters. With Amazon FSx, you can launch and run a file system that provides sub-millisecond access to your data and allows you to read and write data at speeds of up to hundreds of gigabytes per second of throughput and millions of IOPS.
Amazon FSx for Lustre works natively with Amazon S3, making it easy for you process cloud data sets with high performance file systems. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write results back to S3. You can also use FSx for Lustre as a standalone high-performance file system to burst your workloads from on-premises to the cloud. By copying on-premises data to an FSx for Lustre file system, you can make that data available for fast processing by compute instances running on AWS. With Amazon FSx, you pay for only the resources you use. There are no minimum commitments, upfront hardware or software costs, or additional fees.
High performance and scalable
Amazon FSx for Lustre delivers the performance to satisfy a wide variety of high-performance workloads. FSx for Lustre is built on Lustre, a popular high-performance file system that is optimized for data processing, with sub-millisecond latencies and throughput that scales to hundreds of gigabytes per second.
Seamless access to S3 or on-premises data
Amazon FSx works natively with Amazon S3, making it easy to process your S3 data with a high-performance POSIX interface. When linked to an S3 bucket, FSx for Lustre transparently presents S3 objects as files. FSx for Lustre lets you write changed and new data on the file system back to your S3 bucket at any time. FSx for Lustre also enables you to burst your on-premises workloads to to the cloud using Amazon Direct Connect or VPN.
Amazon FSx is fully managed, making it easy to launch and run high-performance file systems in the cloud. You no longer need to worry about hardware provisioning and maintenance, software configuration, and complex performance tuning of file systems. In minutes, you can create and launch a Amazon FSx file system by using the AWS Management Console, the AWS CLI, or an AWS SDK.
Amazon FSx for Lustre helps you cost-optimize your storage for high-performance workloads: It provides cheap and performant storage for processing data, with your long-term data stored in Amazon S3 or other low-cost, long-term data stores. The high-performance of FSx for Lustre allows you to run data processing workloads quicker, reducing time and money spent on compute resources. With FSx for Lustre, you pay only for the resources you use. There are no minimum commitments or upfront fees.
Native file system interface
Amazon FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make any changes. FSx for Lustre provides a native file system interface and works as any file system does with your Linux operating system. It also provides read-after-write consistency and supports file locking. You can control access to your FSx for Lustre file systems with POSIX permissions and Amazon Virtual Private Cloud (VPC) permissions.
Secure and compliant
Amazon FSx automatically encrypts your data-at-rest. If you are subject to regulatory compliance, FSx for Lustre is PCI-DSS, ISO, and SOC compliant, and HIPAA eligible. Amazon FSx supports Amazon Virtual Private Cloud (VPC), so you can launch your FSx for Lustre file system resources in your virtual network.
How it works
Machine learning workloads use massive amounts of training data. These workloads often use shared file storage because multiple compute instances need to process the training datasets concurrently. Amazon FSx is optimal for machine learning workloads, because it provides shared file storage with high throughput and consistent, low latencies to process the ML training datasets.
High performance computing (HPC)
High performance computing (HPC) enables scientists and engineers to solve complex, compute-intensive problems. HPC workloads, like oil & gas discovery and genomics, process massive amounts of data that need to be accessed by multiple compute instances with high levels of throughput. Amazon FSx is ideal for HPC workloads because it provides a file system that’s optimized for the performance and costs of short-lived, high-performance workloads, with file system access across thousands of EC2 instances.
Media processing and transcoding
Media data processing workflows, like video rendering, visual effects, and media production, need the compute and storage resources to handle the massive amounts of data being created. Amazon FSx provides the high performance and low latencies needed for processing, distributing, and analyzing digital media files.
Customers developing autonomous vehicle systems often test models by running simulations and training on massive amounts of vehicle sensor and camera data to ensure vehicle safety. Amazon FSx for Lustre enables you to access that data simultaneously from hundreds of nodes via a standard file system at sub millisecond latency, hundreds of gigabytes per second of throughput, and thousands of IOPS. FSx for Lustre lets you run simulations at scale, achieving thousands of simulated runs per week and accelerating model development.
Big data analytics
Big data analytics use cases, including fraud detection and financial analysis, produce massive volumes of data that need high-performance storage to power data-intensive applications. Managing the escalating volume of data can be costly and complex. Amazon FSx for Lustre is performance and cost-optimized, resulting in faster time to discovery and value to your organization.
Electronic Design Automation (EDA)
EDA is a common high-performance application used to simulate performance and failures during the design phase of silicon chip production. FSx for Lustre provides the performance and flexibility that enables you to innovate faster, design and verify new products, and scale to meet demand.
Learn about the key features of Amazon FSx for Lustre.
Instantly get access to the AWS Free Tier.
Get started building with Amazon FSx for Lustre in the AWS Console.