A: Amazon FSx for Lustre is a fully managed file system that is optimized for compute-intensive workloads, such as high performance computing machine learning, and media data processing workflows. Many of these applications require the high performance and low latencies of scale-out, parallel file systems. Operating these file systems typically requires specialized expertise and administrative overhead, requiring you to provision storage servers and tune complex performance parameters. With Amazon FSx, you can launch and run a Lustre file system that can process massive data sets at up to hundreds of gigabytes per second of throughput, millions of IOPS, and sub-millisecond latencies.
Amazon FSx for Lustre is seamlessly integrated with Amazon S3, making it easy to link your long-term data sets with your high-performance file systems to run compute-intensive workloads. You can automatically copy data from S3 to FSx for Lustre, run your workloads, and then write results back to S3.
FSx for Lustre also enables you to burst your compute-intensive workloads from on-premises to AWS, by allowing you to access your FSx Lustre file system over Amazon Direct Connect or VPN.
FSx for Lustre helps you cost-optimize your storage for compute-intensive workloads: It provides cheap and performant non-replicated storage for processing data, with your long-term data stored durably in Amazon S3 or other low-cost data stores. With Amazon FSx, you pay for only the resources you use. There are no minimum commitments, upfront hardware or software costs, or additional fees.
Q: What use cases does Amazon FSx for Lustre support?
A: Amazon FSx for Lustre is optimized for short-term, compute-intensive workloads, like high performance computing (HPC), machine learning, big data analytics, electronic design automation (EDA), and media processing workflows. Amazon FSx supports these uses cases by providing high throughput for processing massive amounts of data and performs operations with consistent, sub-millisecond latencies.
Q: How do I get started with Amazon FSx for Lustre?
A: To use Amazon FSx for Lustre, you must have an AWS account. If you do not already have an AWS account, you can sign-up for an AWS account.
Once you have created an AWS account, you can create a file system via the AWS Management Console, the AWS Command Line Interface (AWS CLI), and Amazon FSx API (and various language-specific SDKs).
Q: How do I create a file system with Amazon FSx for Lustre?
A: Creating an Amazon FSx for Lustre file system from the Console, CLI, or API is a simple process. Within minutes, your file system is running and accessible to your compute instances. If you link your filesystem to an S3 data lake, your objects will appear as files and directories as soon as your file system is available.
Q: What instance types and AMIs work with Amazon FSx for Lustre?
A: FSx for Lustre is compatible with the most popular Linux-based AMIs, including Red Hat Enterprise Linux (RHEL), CentOS, SUSE Linux and Ubuntu. With FSx for Lustre, you can mix and match the instance types and Linux AMIs that are connected to a single file system.
Q: How do I access an FSx for Lustre file system from a compute instance?
A: You can access your file system from your compute instances using the open-source Lustre client. Once mounted, you can work with the files and directories in your file system just like you would with a local file system.
Q: How do I manage an FSx for Lustre file system?
A: Amazon FSx is a fully managed service, so all of the file storage infrastructure is managed for you. When you use Amazon FSx, you avoid the complexity of deploying and maintaining complex file system infrastructure.
You can administer a file system via the AWS Management Console, the AWS command-line interface (CLI), or the Amazon FSx API (and various language-specific SDKs). The Console, API, and SDK provide the ability to create and delete file systems, create and edit file system tags, and display detailed information about file systems. If you link your filesystem to an S3 data lake, your objects will appear as files and directories as soon as your system is available.
Q: How does Amazon FS for Lustre work with long-term data repositories?
A: Amazon FSx for Lustre allows you to ingest and process large volumes of file data in a high-performance file system, while periodically writing intermediate results to your data repository. Doing so allows you to restart your workload at any time from the latest data you’ve stored in your data repository. When your workload is done, you can write final results from your file system to your data repository, and delete your file system.
Q: If I have data in S3, how do I access it from Amazon FSx for Lustre to process it?
A: If you have data in S3, you can seamlessly link your Amazon FSx for Lustre file system with a specified S3 bucket, making the data in your Amazon S3 data repository accessible to your file system. Once your file system is created, initially the S3 objects’ names and prefixes will be visible as files and directories.
By default, Amazon S3 objects are only loaded into the file system when first accessed by your applications. If your applications access objects that haven’t yet been loaded into your file system, Amazon FSx for Lustre automatically loads the corresponding objects from Amazon S3. Subsequent reads of these files are served directly out of the file system with low, consistent latencies. You can optionally batch hydrate your Amazon FSx for Lustre file system.
Amazon FSx for Lustre uses parallel data transfer techniques to transfer data from S3 at up to hundreds of GBs/s.
Q: How do I commit data written to my Amazon FSx for Lustre file system back to Amazon S3?
A: At any time, you can commit intermediate results you’ve written to your file system back to your Amazon S3 bucket. Only objects that have changed since import from S3 or that are new are written back. You can choose to commit all changed files, or optionally specify specific files or directories to commit. Each time you commit, Amazon FSx for Lustre only copies data to Amazon S3 that hasn’t already been committed. Amazon FSx for Lustre uses parallel data transfer techniques to commit data at up to hundreds of GBs/s.
Q: If I have data on-premises how do I make it available to Amazon FSx for Lustre to process it?
A: If you have compute-intensive workloads running on-premises and demand for computing capacity spikes, you can cloud burst your workloads to Amazon FSx for Lustre by using Amazon Direct Connect or VPN.
Q: What does Amazon FSx for Lustre do to ensure high availability and durability?
A: Amazon FSx for Lustre is designed for short-term, compute-intensive workloads where your long-term data is stored in a durable data repository such as Amazon S3 or an on-premises data store. Amazon FSx for Lustre is cost-optimized for these workloads by providing high-performance, non-replicated file storage that’s designed to be spun up and down as-needed, and isn’t meant to store durable, long-term data.
Q: How do I monitor my file system’s activity?
A: Amazon FSx for Lustre provides native CloudWatch integration, allowing you to monitor file system health and performance metrics in real time. Example metrics include storage consumed, number of compute instance connections, throughput, and number of file operations per second. You can log all Amazon FSx API calls using AWS CloudTrail.
Q: Is Amazon FSx for Lustre POSIX-compliant?
Q: What regions is Amazon FSx for Lustre available in?
A: Please refer to Regional Products and Services for details of Amazon FSx for Lustre service availability by region.
Q: Does Amazon FSx offer a Service Level Agreement (SLA)?
A: Yes. The Amazon FSx SLA provides for a service credit if a customer's monthly uptime percentage is below our service commitment in any billing cycle.
Scale and performance
Q: What performance can I expect from Amazon FSx for Lustre?
A: Amazon FSx for Lustre file systems scale to hundreds of GB/s of throughput and millions of IOPS. Amazon FSx for Lustre also supports concurrent access to the same file or directory from thousands of compute instances, enabling rapid data checkpointing from application memory to storage (a common technique in HPC). Amazon FSx for Lustre provides consistent, sub-millisecond latencies for file operations.
Q: How does throughput scale with storage capacity?
A: FSx for Lustre provides 200MBps of baseline throughput per TB of storage provisioned. As your file systems scales, your throughput can scale up to hundreds of GB/s.
Q: Can I change my file system’s storage capacity and performance?
A: The size and performance of your file system is determined when you create the file system. Once a file system is created the total provisioned size cannot be changed.
Q: How many instances can connect to a file system?
A: Up to tens of thousands of compute instances.
Q: How many file systems can I create?
A: There is a 100-file system limit per account which can be increased upon request.
Security and compliance
Q: Does Amazon FSx for Lustre support data encryption?
A: Yes. All Amazon FSx for Lustre file systems are encrypted at rest with keys managed by the service. Data is encrypted using an XTS-AES-256 block cipher. Data is automatically encrypted before being written to the file system, and automatically decrypted as it is read. These processes are handled transparently by Amazon FSx for Lustre, so you don’t have to modify your applications.
Q: What access control capabilities does Amazon FSx provide?
A: Every FSx for Lustre resource is owned by an AWS account, and permissions to create or access a resource are governed by permissions policies. You specify the Amazon Virtual Private Cloud (VPC) in which your file system is made accessible, and you control which resources within the VPC have access to your file system using VPC Security Groups. You control who can administer your file system and backup resources (create, delete, etc.) using AWS IAM.
Q: What compliance programs does Amazon FSx support?
A: AWS has the longest-running compliance program in the cloud and are committed to helping customers navigate their requirements. Amazon FSx has been assessed to meet global and industry security standards. It complies with PCI DSS, ISO 9001, 27001, 27017, and 27018), in addition to being HIPAA eligible. That makes it easier for you to verify our security and meet your own obligations. For more information and resources, visit our compliance pages. You can also go to the Services in Scope by Compliance Program page to see a full list of services and certifications.
Pricing and billing
Q: How will I be charged and billed for my use of Amazon FSx for Lustre?
A: You pay only for the resources you use. You are billed based on the storage capacity you provision. For pricing information, please visit the pricing section on the product page. See the Amazon FSx for Lustre pricing page for details.
Q: Do your prices include taxes?
A: Except as otherwise noted, our prices are exclusive of applicable taxes and duties, including VAT and applicable sales tax. For customers with a Japanese billing address, use of AWS services is subject to Japanese Consumption Tax. Learn more.