Scaling HPC workloads by 4x using AWS PCS for Amazon Leo

The Amazon Leo chip design team uses electronic design automation (EDA) software to design and validate its custom satellite chips. This software runs millions of compute jobs to simulate and verify the hardware designs. The team used enterprise-class legacy job scheduling software, but this solution had a static cluster with a fixed node count. It couldn’t scale as the organization grew and the number of simulations expanded by 10 times. Adding capacity involved slow, manual processes for the four-person platform engineering team, which spent about 25 percent of its time managing scheduling operations rather than engineering work. Meanwhile, chip designers often had to wait 4–6 hours with their jobs in the queue, which could get up to 3,000 jobs long on busy days.

Amazon Leo depends on hardware designers to deliver chips before software and other downstream component teams can finalize their work. However, the legacy job scheduling software was behind on updates and slowing down the design process. “It’s very expensive to miss a hardware deadline,” says Ekjot Basra, systems development engineer manager at Amazon Leo. To remove the bottleneck from this critical workflow, Amazon Leo chose to migrate its job scheduling workloads to a fully managed, cloud-based HPC service on AWS: AWS PCS.

Amazon Leo selected AWS PCS to run HPC workloads within its EDA environment because the solution could provide the scalable compute capacity that Amazon Leo needed while being simpler to use and maintain. AWS PCS offers automatic scaling, scaling up and down capacity dynamically to handle the jobs in the queue without overprovisioning when demand is lower. Using the solution, the team can define queues more easily and experiment with node instance sizes. “If we realize that this month most of the jobs coming through require only a certain amount of memory, it’s easy enough to flip compute node groups in AWS PCS and reallocate so it runs more efficiently,” says Basra.

The EDA solution serves as a centralized environment for the massive amounts of shared data that hardware developers need to access across various teams. For high performance storage, the EDA environment stores 1 PB of data using Amazon Elastic File System (Amazon EFS)—serverless, fully elastic file storage—and 500 TB of data using Amazon FSx for NetApp ONTAP, fully managed shared storage built on NetApp’s popular ONTAP file system. The new EDA environment provides not only better HPC performance using AWS PCS but also automated user provisioning and simplified onboarding.

In addition to scalability and performance, Amazon Leo chose AWS PCS, Amazon EFS, and Amazon FSx for NetApp ONTAP to take advantage of fully managed services. “We’ve gotten a taste for managed services on AWS,” says Basra. “It helps with our security and compliance posture if we have to worry less about patching individual instances and operational pieces ourselves versus trusting that teams like AWS PCS maintain that for us.”

Since migrating to AWS PCS, Amazon Leo increased its monthly compute by approximately 4 times, from 300,000 to 1.3 million jobs, while reducing costs by more than 30 percent. Wait times in the job queue reduced from 4–6 hours to near 0, helping designers iterate on chip designs much faster while still meeting deadlines. Operational overhead for the platform engineering team was also reduced by approximately 90 percent. “We worry less about the day-to-day management of this crucial piece of infrastructure, which frees us to innovate more in other areas,” says Basra.

The team can keep the solution updated and patched without downtime for the users, improving its security and compliance posture. The solution scales to accommodate more jobs—increasing capacity by more than four times—without the platform engineering team needing to police resources or manually prioritize certain jobs over others. And the infrastructure can handle increased workload as Amazon Leo implements AI tools to iterate faster. “AWS PCS can handle burst load at various points in the project, and that ability to adapt is crucial for us,” says Basra.

Going forward, Amazon Leo will continue to build out best practices and monitoring for its AWS PCS environment to optimize how it uses the solution. Because chip quality and iteration speed depend directly on how well the compute cluster performs, a more effective AWS PCS environment helps Amazon Leo deliver more capable satellite chips faster. “The more effective we are in our compute cluster, the better the quality of the chips that will come out and the faster we can iterate,” says Basra. “Using AWS PCS is the way we make sure everything is running as smoothly as possible as we build more features into the next generation of chips.”