Toyota Research Institute chooses FSx for Lustre to reduce object recognition machine learning training times.
Toyota Research Institute (TRI) collects and processes large amounts of sensor data from their autonomous vehicles (AV) test drives. Each training data set is staged in an on-premises NAS device and transferred to Amazon Simple Storage Service (Amazon S3) before processing on a powerful GPU compute cluster. TRI needed a high-performance file system to pair with their compute resources, speed up their ML model training, and accelerate insights for their data scientists.
“We needed a parallel file system for our ML training data sets and chose Amazon FSx for Lustre for its higher availability and durability, compared to our legacy file system offering. The integration with AWS services, including S3, also made it the preferred option for our high performance file storage.”
David Fluck, Software Engineer - Toyota Research Institute
INEOS TEAM UK accelerates boat design for America’s Cup using AWS.
Challenge: Formed in 2018, INEOS TEAM UK aims to bring the America’s Cup—the oldest international sporting trophy in the world—to Great Britain. America’s Cup restricts on-water testing to no more than 150 days prior to the event, so high-performance computational fluid dynamics (CFD) simulations of monohulls and foils become key to a winning boat design.
Solution: Using AWS, INEOS TEAM UK can process thousands of design simulations for its America’s Cup boat in one week versus in more than a month using an on-premises environment. INEOS TEAM UK will compete in the 36th edition of the America’s Cup in 2021. The team is using an HPC environment running on Amazon EC2 Spot Instances. To ensure fast disk performance for the thousands of simulations completed each week, the team also used Amazon FSx for Lustre to provide a fast, scalable, and secure high-performance file system based on Amazon Simple Storage Service (Amazon S3).
"AWS allows us to take bigger design steps, simply because we have more time to understand our results."
Nick Holroyd, Head of Design - INEOS TEAM UK
T-Mobile realizes $1.5M in annual savings and doubles the speed of SAS Grid workloads using Amazon FSx for Lustre.
Challenge: T-Mobile was experiencing high management overhead and performance difficulties with their self-managed SAS Grid workload.
Solution: T-Mobile deployed Amazon FSx for Lustre, a fully managed high-performance file system, to migrate and scale their SAS Grid infrastructure. T-Mobile utilized the tight integration of Amazon FSx and S3 to reduce their storage overhead and optimize operations.
"Amazon FSx for Lustre helped us double the speed of our SAS Grid workloads, reduce our Total Cost of Ownership by 83% and completely eliminate our operational burden. Partnering with AWS enables us to focus on what we do best, developing innovative products for our customers, while relying on the cutting-edge storage features of FSx, and world-class hosting capabilities of AWS."
Dinesh Korde, Senior Manager Software Development - T-Mobile
Hive VFX cuts upfront studio costs, operates as cloud VFX studio on AWS.
Challenge: Hive needed high-performance infrastructure to launch a small, independent cloud studio for remote artists around the world to create quality content.
Solution: Fully managed Amazon FSx for Lustre, integrated with Amazon S3, provided fast access to AWS compute resources without a big upfront investment or in-house IT team expertise. The seamless synchronization of file data and file permissions between FSx Lustre and S3 enabled Hive VFX to store a large volume of images and share project data across continents.
"I can spin up an Amazon FSx for Lustre file system in 5 minutres and it's all managed by AWS."
Bernie Kimbacher, Founder - Hive VFX
Maxar uses AWS to deliver forecasts 58% faster than weather supercomputer.
Challenge: Maxar Technologies, a trusted partner and innovator in Earth intelligence and Space infrastructure, needed to deliver weather forecasts faster compared to its on-premises supercomputer.
Solution: Maxar worked with AWS to create an HPC solution with key technologies including Amazon Elastic Compute Cloud (Amazon EC2) for secure, highly reliable compute resources, Amazon FSx for Lustre to accelerate the read/write throughput of its application, and AWS ParallelCluster to quickly build HPC compute environments on AWS.
"Maxar used Amazon FSx for Lustre in our AWS HPC solution for running NOAA's numerical weather forecasting model. This allowed us to reduce compute time by 58%, generating the forecast in about 45 minutes for a much more cost-effective price point. Maximizing our AWS compute resources was an incredible performance boost for us."
Stefan Cecelski, PhD, Senior Data Scientist & Engineer - Maxar Technologies
Lyell accelerates their cell-based cancer treatment research with Amazon FSx for Lustre.
Challenge: Lyell delivers curative, cell-based cancer treatments that require running large scale computational design of proteins. These workloads were traditionally run on premises, but the company needed a more scalable, cost-effective solution as they were limited to running only one experiment per month.
Solution: Since migrating their file system to FSx for Lustre, data scientists can spin-up and spin-down thousands of HPC clusters consisting of EC2 instances and Amazon FSx file systems, enabling them to run processing-heavy experiments rapidly, and only pay for compute and storage for the duration of the workload.
"Amazon for FSx Lustre speeds up our research in developing the next generation cancer treatment. Using FSx, we have reduced the execution time of our experiments from weeks down to hours, and enabled scientists to test many more hypotheses than before. Our workloads running on tens of thousands of compute nodes, can now use FSx to access S3 data at super-high sets."
Anish Kejariwal, Head of Data Analytics Engineering - Lyell Immunopharma
BlackThorn Therapeutics accelerates time-to-insight with FSx for Lustre.
Challenge: Processing magnetic imaging (MRI) data using standard DiY cloud file systems was resource and time-intensive. BlackThorn needed a compute-intensive, shared file storage solution to help simplify their data science and machine learning workflows.
Solution: Amazon FSx for Lustre is integrated with Amazon S3 and Amazon SageMaker, providing fast processing for their ML training data sets as well as seamless access to compute using Amazon EC2 instances.
"FSx for Lustre has enabled us to create a high-performance MRI data processing pipeline. Data processing time for our ML-based workflows was cut down to minutes compared to days and weeks."
Oscar Rodriguez, Senior Director, Innovation & Technology - BlackThorn Therapeutics
Conductor Technologies accelerates rendering workloads by up to 30% using Amazon FSx for Lustre.
Challenge: Conductor, makers of the cloud rendering platform behind blockbuster theatrical films such as Deadpool and Star Trek Beyond, was faced with scaling and efficiency issues running file systems with their previous cloud provider. The company built a self-managed file system to relieve latency and meet their customers’ expected SLAs, but needed an easily-deployed managed service in order to scale their business.
Solution: Using Amazon FSx for Lustre and native integration with Amazon S3, the company was able to reduce their render times and eliminate operational overhead of managing their in-house file storage solution in the cloud.
"In our recent migration effort to AWS, we’ve chosen to use FSx for Lustre to supercharge our customers' VFX jobs on AWS. Initial testing has shown as much as a 30% reduction in spin-up and runtimes over our self-managed file system. In addition, we’re looking to leverage FSx for Lustre with Amazon SageMaker to provide our customers with predictions about the compute capacity and render times for their jobs, allowing them to make decisions about optimizing spend with Amazon EC2 Spot instances. AWS has made it faster and cheaper for studios to render and deliver an even more cutting-edge effects for tomorrow’s biggest films."
Mac Moore, CEO - Conductor Technologies
Qubole improves data durability while reducing cost with Amazon FSx for Lustre.
Challenge: Qubole was seeking a high-performance storage solution to process analytical and AI/ML workloads for their customers. They needed to easily store and process the intermediate data held in their EC2 Spot Fleet.
Solution: Qubole used Amazon FSx for Lustre to store and process intermediate data through its parallel, high-speed file system.
"Our users’ two biggest problems, high costs and intermediate data loss, stemmed from using idle EC2 instances and EC2 Spot instances to process and store intermediate data generated by distributed processing frameworks like Hive and Spark. We were able to solve this problem by using Amazon FSx for Lustre, a highly performant file system, to offload intermediate data. Now our users do not have to pay to maintain idle instances and are not affected by interrupted EC2 Spot nodes. Amazon FSx helped our users reduce total costs by 30%."
Joydeep Sen Sarma, CTO - Qubole
Learn about the key features of Amazon FSx for Lustre.
Instantly get access to the AWS Free Tier.
Get started building with Amazon FSx for Lustre in the AWS Console.