Amazon FSx for Lustre provides a fully managed high-performance Lustre file system that allows file-based applications to access data with hundreds of gigabytes per second of data, millions of IOPS, and sub millisecond latencies. Amazon FSx works natively with Amazon S3, letting you transparently access your S3 objects as files on Amazon FSx to run analyses for hours to months. You can then write results back to S3, and simply delete your file system. FSx for Lustre also enables you to burst your data processing workloads from on-premises to AWS, by allowing you to access your FSx file system over Amazon Direct Connect or VPN.
Lustre file system
Lustre is a popular open-source parallel file system that is designed for high-performance workloads. These workloads include HPC, machine learning, analytics, and media processing.
A parallel file system provides high throughput for processing large amounts of data and performs operations with consistently low latencies. It does so by storing data across multiple networked servers that thousands of compute instances can interact with concurrently. The Lustre file system provides a POSIX-compliant file system interface.
Low latencies and high throughput
Amazon FSx can scale up to hundreds of gigabytes per second of throughput, and millions of IOPS. Amazon FSx provides high throughput for processing large amounts of data and performs operations with consistent, sub-millisecond latencies. Amazon FSx for Lustre supports file access to thousands of EC2 instances, enabling you to provide file storage for your high-performance workloads, like genomics, seismic exploration, and video rendering.
Seamless access to your data
Amazon FSx works natively with Amazon S3, making it easy to access your S3 data to run data processing workloads. Your S3 objects are presented as files in your file system, and you can write your results back to S3. You can also copy your on-premises data to FSx for Lustre using Amazon Direct Connect and VPN. This lets you run data processing workloads on FSx for Lustre and store your long-term data on S3 or on-premises data stores.
Amazon FSx provides automated integration that makes it easy to access data from S3. With a few clicks in console, you can create a file system that’s linked to an S3 bucket you specify. After you link your S3 bucket to your file system, FSx for Lustre transparently presents S3 objects as files and allows you to write results back to S3. For file systems linked to S3 buckets, Amazon FSx maintains a one-to-one mapping between your S3 objects and Amazon FSx files.
At any time, you can send results written to your Amazon FSx file system back to your S3 bucket. You can choose to commit all writes, or specific files or directories. Each time you export your results, Amazon FSx only copies incremental data to S3, and the results are written to a separate part of your S3 bucket. Amazon FSx for Lustre uses parallel data-transfer techniques to commit data back to S3.
You can use Amazon FSx for Lustre for on-premises workloads that need to burst to the cloud due to peak demands or capacity limits. To move your existing on-premises data into Amazon FSx, you can mount your Amazon FSx for Lustre file system from an on-premises client over AWS Direct Connect or VPN, and then use parallel copy tools to import your data to your Amazon FSx for Lustre file system. At any time you can write your results back to be durably stored in your data lake.
Storage for high-performance workloads
FSx for Lustre is optimized for a variety of data processing workloads that run from hours to months. These workloads require high levels of throughput, IOPS, and consistent low-latencies. These data processing workloads often rely on parallel file systems to get the scale-out performance they depend on. Amazon FSx is optimized for the performance and costs of high-performance workloads with your long-term storage on S3 or on-premises data stores.
With a few clicks in the Amazon FSx console, CLI, or API you can create and launch a Lustre file system that is accessible to your compute instances and linked to your S3 bucket.
Automated operations: Amazon FSx fully manages and maintains the infrastructure and software powering your file system, and eliminates the need for administrative overhead and ongoing maintenance. Amazon FSx performs routine Lustre updates, and detects and addresses hardware issues.
Security and compliance
All Amazon FSx file system data is encrypted at rest.
AWS has the longest-running compliance program in the cloud and is committed to helping customers navigate their requirements. Amazon FSx has been assessed to meet global and industry security standards. It complies with PCI DSS, ISO 9001, 27001, 27017, and 27018), and SOC 1, 2, and 3, in addition to being HIPAA eligible. That makes it easier for you to verify our security and meet your own obligations. For more information and resources, visit our compliance pages. You can also go to the Services in Scope by Compliance Program page to see a full list of services and certifications.
You can access your file system from your compute instances using the open-source Lustre client. Once mounted, you can work with the files and directories in your file system just like you would with a local file system. FSx for Lustre is compatible with the most popular Linux-based AMIs, including Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, Ubuntu, and SUSE Linux. With FSx for Lustre, you can mix and match the instance types and Linux AMIs that are connected to a single file system. FSx for Lustre also enables you to burst your high-performance workloads from on-premises to AWS, by allowing you to access your FSx file system over Amazon Direct Connect or VPN. With AWS Direct Connect or VPN, you can use parallel copy tools to move on-premises data to your Amazon FSx file system.
You access your Amazon FSx file system from endpoints in your Amazon VPC, which enables you to isolate your file system in your own virtual network. You can configure security group rules and control network access to your Amazon FSx file systems.
Amazon FSx is integrated with AWS Identity and Access Management (IAM). This integration means that you can control the actions your AWS IAM users and groups can take to manage your file systems (such as creating and deleting file systems). You can also tag your Amazon FSx resources and control the actions that your IAM users and groups can take based on those tags.
FSx for Lustre provides a cost-effective and performant file system for processing data, with your long-term data stored durably in Amazon S3 or other low-cost data stores. You pay only for the resources you use, with no minimum commitments or upfront fees. You are billed in per-second increments based on the storage capacity that you provision. Amazon FSx for Lustre reduces the storage costs for you to run high-performance workloads by enabling you to launch and terminate file systems as needed and store long-term data in Amazon S3. For more details, see the Amazon FSx pricing page.