Toyota Research Institute chooses FSx for Lustre to reduce object recognition machine learning training times.
Toyota Research Institute (TRI) collects and processes large amounts of sensor data from their autonomous vehicles (AV) test drives. Each training data set is staged in an on-premises NAS device and transferred to Amazon S3 before processing on a powerful GPU compute cluster. TRI needed a high-performance file system to pair with their compute resources, speed up their ML model training, and accelerate insights for their data scientists.
“We needed a parallel file system for our ML training data sets and chose Amazon FSx for Lustre for its higher availability and durability, compared to our legacy file system offering. The integration with AWS services, including S3, also made it the preferred option for our high performance file storage.”
David Fluck, Software Engineer - Toyota Research Institute
This Is My Architecture videos: TRI large-scale and high-performance distributed training platform for data scientists.
T-Mobile realizes $1.5M in annual savings and doubles the speed of SAS Grid workloads using Amazon FSx for Lustre.
Challenge: T-Mobile was experiencing high management overhead and performance difficulties with their self-managed SAS Grid workload.
Solution: T-Mobile deployed Amazon FSx for Lustre, a fully managed high-performance file system, to migrate and scale their SAS Grid infrastructure. T-Mobile utilized the tight integration of Amazon FSx and S3 to reduce their storage overhead and optimize operations.
"Amazon FSx for Lustre helped us double the speed of our SAS Grid workloads, reduce our Total Cost of Ownership by 83% and completely eliminate our operational burden. Partnering with AWS enables us to focus on what we do best, developing innovative products for our customers, while relying on the cutting-edge storage features of FSx, and world-class hosting capabilities of AWS."
Dinesh Korde, Senior Manager Software Development - T-Mobile
Lyell accelerates their cell-based cancer treatment research with Amazon FSx for Lustre.
Challenge: Lyell delivers curative, cell-based cancer treatments that require running large scale computational design of proteins. These workloads were traditionally run on premises, but the company needed a more scalable, cost-effective solution as they were limited to running only one experiment per month.
Solution: Since migrating their file system to FSx for Lustre, data scientists can spin-up and spin-down thousands of HPC clusters consisting of EC2 instances and Amazon FSx file systems, enabling them to run processing-heavy experiments rapidly, and only pay for compute and storage for the duration of the workload.
"Amazon for FSx Lustre speeds up our research in developing the next generation cancer treatment. Using FSx, we have reduced the execution time of our experiments from weeks down to hours, and enabled scientists to test many more hypotheses than before. Our workloads running on tens of thousands of compute nodes, can now use FSx to access S3 data at super-high sets."
Anish Kejariwal, Head of Data Analytics Engineering - Lyell Immunopharma
Conductor Technologies accelerates rendering workloads by up to 30% using Amazon FSx for Lustre.
Challenge: Conductor, makers of the cloud rendering platform behind blockbuster theatrical films such as Deadpool and Star Trek Beyond, was faced with scaling and efficiency issues running file systems with their previous cloud provider. The company built a self-managed file system to relieve latency and meet their customers’ expected SLAs, but needed an easily-deployed managed service in order to scale their business.
Solution: Using Amazon FSx for Lustre and native integration with Amazon S3, the company was able to reduce their render times and eliminate operational overhead of managing their in-house file storage solution in the cloud.
"In our recent migration effort to AWS, we’ve chosen to use FSx for Lustre to supercharge our customers' VFX jobs on AWS. Initial testing has shown as much as a 30% reduction in spin-up and runtimes over our self-managed file system. In addition, we’re looking to leverage FSx for Lustre with Amazon SageMaker to provide our customers with predictions about the compute capacity and render times for their jobs, allowing them to make decisions about optimizing spend with Amazon EC2 Spot instances. AWS has made it faster and cheaper for studios to render and deliver an even more cutting-edge effects for tomorrow’s biggest films."
Mac Moore, CEO - Conductor Technologies
Qubole improves data durability while reducing cost with Amazon FSx for Lustre.
Challenge: Qubole was seeking a high-performance storage solution to process analytical and AI/ML workloads for their customers. They needed to easily store and process the intermediate data held in their EC2 Spot Fleet.
Solution: Qubole used Amazon FSx for Lustre to store and process intermediate data through its parallel, high-speed file system.
"Our users’ two biggest problems, high costs and intermediate data loss, stemmed from using idle EC2 instances and EC2 Spot instances to process and store intermediate data generated by distributed processing frameworks like Hive and Spark. We were able to solve this problem by using Amazon FSx for Lustre, a highly performant file system, to offload intermediate data. Now our users do not have to pay to maintain idle instances and are not affected by interrupted EC2 Spot nodes. Amazon FSx helped our users reduce total costs by 30%."
Joydeep Sen Sarma, CTO - Qubole
Learn about the key features of Amazon FSx for Lustre.
Instantly get access to the AWS Free Tier.
Get started building with Amazon FSx for Lustre in the AWS Console.