Amazon FSx for Lustre provides a fully managed Lustre file system that allows file-based applications to access data with hundreds of gigabytes per second of data, millions of IOPS, and sub millisecond latencies. Amazon FSx provides high-performance storage at a low cost, because it uses nonreplicated SSD-based storage, with long-term data stored on lower-cost S3 or on-premises data stores. Amazon FSx is seamlessly integrated with Amazon S3, letting you automatically copy S3 data into Amazon FSx to run analyses for hours to weeks. You can then write results back to S3, and simply delete your file system. FSx for Lustre also enables you to burst your compute-intensive workloads from on-premises to AWS, by allowing you to access your FSx file system over Amazon Direct Connect or VPN.
Lustre file system
Lustre is a popular open-source parallel file system that is designed for compute-intensive workloads. These workloads include high performance computing, machine learning, and media processing.
A parallel file system provides high throughput for processing massive amounts of data and performs operations with consistently low latencies. It does so by storing data across multiple networked servers that thousands of compute instances can interact with concurrently. The Lustre file system provides a POSIX-compliant file system interface.
Massively scalable performance
Amazon FSx can scale to hundreds of gigabytes per second of throughput, and millions of IOPS. Amazon FSx provides high throughput for processing massive amounts of data and performs operations with consistent, sub-millisecond latencies. Amazon FSx for Lustre supports file access to thousands of EC2 instances, enabling you to provide file storage for your compute-intensive workloads, like genomics, seismic exploration, and video rendering.
Seamless access to your data
Amazon FSx is integrated with Amazon S3, making it easy to access your S3 data to run compute-intensive workloads. Your S3 data is automatically copied to FSx for Lustre, and you can write your results back to S3. You can also copy your on-premises data to FSx for Lustre using Amazon Direct Connect and VPN. This lets you run compute-intensive workloads on FSx for Lustre and store your long-term data on S3 or on-premises data stores.
Amazon FSx provides automated integration that makes it easy to copy data from S3. With a few clicks in console, you can create a file system that’s linked to an S3 bucket you specify. After you link your S3 bucket to your system, Amazon FSx automatically copies the data you request from S3 to your file system in real time. It works as your applications access the data. For file systems linked to S3 buckets, Amazon FSx maintains a one-to-one mapping between your S3 objects and Amazon FSx files.
At any time, you can send results written to your Amazon FSx file system back to your S3 bucket. You can choose to commit all writes, or specific files or directories. Each time you export your results, Amazon FSx only copies incremental data to S3, and the results are written to a separate part of your S3 bucket. Amazon FSx for Lustre uses parallel data-transfer techniques to commit data back to S3.
You can use Amazon FSx for Lustre for on-premises workloads that need to burst to the cloud due to peak demands or capacity limits. To move your existing on-premises data into Amazon FSx, you can mount your Amazon FSx for Lustre file system from an on-premises client over AWS Direct Connect or VPN, and then use parallel copy tools to import your data to your Amazon FSx for Lustre file system. At any time you can write your results back to be durably stored in your data lake.
Storage for compute-intensive workloads
Compute-intensive workloads are high-performance jobs that run from hours to weeks. These workloads require extremely high levels of throughput, IOPS, and consistent low-latencies. Compute intensive workloads often rely on parallel file systems to get the scale-out performance they depend on. Amazon FSx is optimized for the performance and costs of compute-intensive workloads, because it provides fully managed Lustre parallel file systems, on non-replicated storage, with your long-term storage on S3 or on-premises data stores.
With a few clicks in the Amazon FSx console, CLI, or API you can create and launch a Lustre file system that is accessible to your compute instances and linked to your S3 bucket.
Automated operations: Amazon FSx fully manages and maintains the infrastructure and software powering your file system, and eliminates the need for administrative overhead and ongoing maintenance. Amazon FSx performs routine Lustre updates, and detects and addresses hardware issues.
Security and compliance
All Amazon FSx file system data is encrypted at rest.
AWS has the longest-running compliance program in the cloud and is committed to helping customers navigate their requirements. Amazon FSx has been assessed to meet global and industry security standards. It complies with PCI DSS, ISO 9001, 27001, 27017, and 27018), in addition to being HIPAA eligible. That makes it easier for you to verify our security and meet your own obligations. For more information and resources, visit our compliance pages. You can also go to the Services in Scope by Compliance Program page to see a full list of services and certifications.
You can access your file system from your compute instances using the open-source Lustre client. Once mounted, you can work with the files and directories in your file system just like you would with a local file system. FSx for Lustre is compatible with the most popular Linux-based AMIs, including Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux. With FSx for Lustre, you can mix and match the instance types and Linux AMIs that are connected to a single file system. FSx for Lustre also enables you to burst your compute-intensive workloads from on-premises to AWS, by allowing you to access your FSx file system over Amazon Direct Connect or VPN. With AWS Direct Connect or VPN, you can use parallel copy tools to move on-premises data to your Amazon FSx file system.
You access your Amazon FSx file system from endpoints in your Amazon VPC, which enables you to isolate your file system in your own virtual network. You can configure security group rules and control network access to your Amazon FSx file systems.
Amazon FSx is integrated with AWS Identity and Access Management (IAM). This integration means that you can control the actions your AWS IAM users and groups can take to manage your file systems (such as creating and deleting file systems). You can also tag your Amazon FSx resources and control the actions that your IAM users and groups can take based on those tags.
FSx for Lustre helps you cost-optimize your storage for compute-intensive workloads: It provides cheap and performant non-replicated storage for processing data, with your long-term data stored durably in Amazon S3 or other low-cost data stores. You pay only for the resources you use, with no minimum commitments or upfront fees. You are billed in per-second increments based on the storage capacity that you provision. Amazon FSx for Lustre reduces the storage costs for you to run data-intensive workloads by enabling you to launch and terminate file systems as needed and store their long-term data in Amazon S3. For more details, see the Amazon FSx pricing page.