Accelerating Car Design in a Sustainable and Flexible Environment Using HPC on AWS with TGR-E

Based in Cologne, Germany, TGR-E is a cutting-edge engineering center and subsidiary of Toyota Motor Corporation (Toyota). It focuses on motorsport research and development for World Endurance Championship and World Rally Championship races, including the 24 Hours of Le Mans. Previously, the company ran its simulations using an on-premises HPC cluster, using only 60 percent of its resources on average but paying for 100 percent. It also encountered insufficient compute capacity during peak periods, especially before large events, causing engineers to wait for compute resources and delaying engineering results. With the on-premises cluster lease set to expire, TGR-E turned to AWS to run its HPC workloads.

“The idea was to follow the enterprise strategy of Toyota, find a solution in the cloud, and pay for only what we use,” says Oliver Meis, senior manager at TGR-E. “Although we averaged 60 percent usage for our compute resources, we still had peaks where even 100 percent capacity was not enough. By migrating to AWS, we could scale to cover these peaks as well.”

To support its migration from on premises to the cloud, TGR-E participated in the AWS Migration Acceleration Program (AWS MAP), a comprehensive and proven cloud migration program based on the AWS team’s experience migrating thousands of enterprise customers to the cloud. The company completed AWS MAP working alongside Eviden, an Atos business and AWS Partner that offers IT services and solutions for the efficient use of complex computer environments in research, development, and computation.

After testing different instance types, TGR-E migrated its on-premises HPC cluster to the HPC Optimized instance family of Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload. These instances include Amazon EC2 Hpc6a Instances, which are powered by third-generation AMD EPYC processors; Amazon EC2 Hpc7a Instances, which are powered by fourth-generation AMD EPYC processors; and Amazon EC2 Hpc6id Instances, which are powered by third-generation Intel Xeon Scalable processors. By adopting HPC Optimized instances, TGR-E gained the computational power required to run complex simulations efficiently at scale. It uses HPC on AWS to run multiple computer-aided engineering simulations using a variety of open-source and commercial packages in computational fluid dynamics, dynamic load analysis, finite element modeling, and more.

TGR-E created instantly deployable HPC architectures for application software with remote desktops based on NICE DCV, a high-performance remote display protocol. This approach combines all open-source and commercial software accessed by engineering teams in a single environment on AWS. These simulations help engineers understand the behavior and interaction of liquids and gases with surfaces, which is crucial for optimizing the aerodynamics, cooling, and overall performance of motorsport vehicles.

TGR-E also adopted AWS ParallelCluster, an open-source cluster management tool that simplifies the deployment and management of HPC clusters. Its engineers use this tool to automate the provisioning of resources for HPC applications; they submit simulation parameters to the job scheduler, and AWS ParallelCluster handles the rest, including the scaling up and down of Amazon EC2 compute nodes.

The solution accelerates HPC jobs by attaching Amazon EC2 instances to Elastic Fabric Adapter (EFA), a network interface for Amazon EC2 instances that provides low network latency and 300 Gbps of network bandwidth. As a result, MPI jobs using EFA have a lower latency compared to the traditional on-premises network interface without any additional cost.

To store and protect data, TGR-E uses Amazon Simple Storage Service (Amazon S3), an object storage service, alongside Amazon FSx for Lustre, which provides fully managed shared storage with the scalability and performance of the popular Lustre file system. Amazon FSx for Lustre handles the company’s most input- and output-intensive workloads. When linked to an Amazon S3 bucket, an Amazon FSx for Lustre file system presents Amazon S3 objects as files and lets engineers write results back to Amazon S3.

Since migrating to AWS, TGR-E has seen numerous benefits. From day one, the company has experienced a 10 percent increase in processing speed along with consistent and reliable performance. It has experienced no downtime or job slowdowns, leading to more efficient and robust workflows. Additionally, the cost benefits have exceeded expectations, with TGR-E unlocking significant savings by omitting traditional data center costs, such as rack space, cooling, and energy.

By taking advantage of HPC on AWS, TGR-E is accelerating motorsport car design to create better vehicles. TGR-E has also reduced the carbon footprint of its HPC workloads by 95% compared with on premises based on the AWS customer carbon footprint tool, which uses simple-to-understand data visualizations to help customers review, evaluate, and forecast emissions. TGR-E plans to continue its digital transformation, focusing on efficiency as a way to reduce emissions. By using advanced AWS technology, TGR-E optimizes its HPC applications while supporting the company’s long-term strategic vision to develop carbon neutral mobility solutions. The process creates a flywheel effect as TGR-E continually brings new simulation workloads onto AWS and benefits from new Amazon EC2 instances to optimize current workflows. To optimize its HPC workloads even further, the company plans to explore AWS Graviton processors, which are designed to deliver the best price performance for cloud workloads running on Amazon EC2.

“With HPC on AWS, we have much more flexibility to react to future demands or even surprise peaks,” says Meis. “But the story won’t end there. We now have the potential for continual improvement on AWS, and we can benefit from new technologies much sooner than we could on premises.”