Unlock large-scale autonomous driving simulations on AWS with IPG

This post was contributed by Ryan Qi and Nikita Buldakov from AWS, and David Howarth and Pravarshi Kolluri fro IPG Automotive

Autonomous vehicle and advanced driver assistance systems (AV/ADAS) are transforming the automotive industry to deliver safer, more efficient, and more convenient vehicles. We see this transition in features like adaptive cruise control, automatic emergency braking, and various levels of autonomous driving. Developing these features requires millions of miles of rigorous testing to make sure that vehicles correctly perceive their environment, make appropriate decisions, and perform reliably under varying road conditions.

Manufacturers can’t rely solely on the confines of the real world for experiments at this scale, so they’re increasingly using simulations as a cost-effective sandbox that can scale to millions of scenarios, including some rare edge-cases and dangerous situations.

These are complex, so the software often combines several commercial simulators and tests many interdependent vehicle systems like sensors, perception algorithms, actuators, vehicle dynamics, and more. The components have separate (and evolving) dependencies making it really challenging to merge all of them into a single monolithic Docker container. And in a normal organization each of these components are probably developed by different teams – making a monolithic container prohibitively time consuming to prototype, debug, and test. All of this highlighted the need for a more advanced cloud infrastructure that worked for AV/ADAS development.

In this post, we’ll explain how to run large-scale AV/ADAS simulations using the popular IPG CarMaker simulator user interface using AWS Batch’s multi-container features.

Building multi-container autonomous driving simulations

IPG Automotive is an AWS Partner who have been developing software for decades. They’re best known for their AV/ADAS simulators: CarMaker, TruckMaker and MotorcycleMaker. IPG CarMaker (as shown in Figure 1) provides a high-fidelity, scalable virtual vehicle model with a strong correlation to real-world driving. It supports flexible scenario generation and detailed sensor simulation. These allow for comprehensive testing and validation of autonomous functions under a wide range of conditions.

CarMaker’s open integration and test platform facilitates the seamless incorporation of both virtual and real components into a virtual prototype, enhancing the depth and accuracy of simulations. By running in the cloud, CarMaker can leverage advanced computational resources, enabling efficient scaling and cost optimization. This means faster development times, reduced need for physical prototypes, and the ability to conduct reliable system pre-development tests and user acceptance tests.

Figure 1 – AV/ADAS simulations with multiple scenarios using IPG CarMaker

IPG users previously faced a limitation of single-container jobs when using AWS Batch for AV/ADAS simulations. This meant that engineers had to undertake the cumbersome task of combining all simulation components into one large, complex monolithic container. The process was time-consuming and complicated collaboration across teams. Making code changes was daunting, because it required a complete rebuild of the monolithic container and complicating development, operations, and debugging.

The recent introduction of multi-container jobs by AWS Batch presents a solution to these challenges.

This allows IPG users to conduct large-scale simulations for complex systems in autonomous vehicles way more efficiently. With multi-container jobs, engineers now work with multiple containers simultaneously, streamlining the development process by eliminating the need to merge everything into a monolithic container or by developing in-house multi-container tools. They can use AWS Batch’s sophisticated scaling, scheduling, and cost management features, getting a smoother workflow and a more collaborative environment across teams.

In the rest of this post, we’ll show how to set up and execute an AV/ADAS simulation using AWS Batch and the IPG CarMaker user interface. Figure 2 shows that the simulation leverages seven containers for each Batch job. One of these containers hosts the headless CarMaker simulator, while the remaining six represent the individual ADAS features, like Auto Emergency Braking, and Adaptive Cruise Control and so forth – with different testing scenarios, each running a ROS2 environment. This also allows you to scale the simulation across a vast number of testing scenarios by creating multiple Batch jobs (Job 1, Job 2, …, Job N).

Figure 2 – Architecture for running multi-container jobs with IPG and AWS Batch

Run simulations on AWS Batch with IPG CarMaker

Step 1 – setup your AWS infrastructure

To do this, you’ll first need to create the necessary AWS resources. You can use Scenario Manager, CarMaker’s graphical interface, to setup AWS architecture. Select Setup AWS Architecture from the navigation panel. There you can select a cloud formation template to create your AWS resources.

CarMaker comes with a default AWS CloudFormation template that will setup key components of your AWS architecture:

1. Amazon Simple Storage Service (Amazon S3) buckets to store startup files and simulation results.
2. Networking configurations with Amazon Virtual Private Cloud (VPC) including route tables and security groups.
3. AWS Batch resources including Compute Environment, Job Queue, and Job Definition to schedule your simulations and manage the underlying compete infrastructure.
4. Amazon Elastic Container Registry (Amazon ECR) to store your Docker images.

Figure 3 – Use CarMaker Scenario Manager to setup your AWS architecture

You can modify CarMaker’s default launch template to suite your requirements, like Security Groups, VPCs, or Amazon S3 buckets – or you can bring your own launch template instead.

Step 2 – create CarMaker scenarios

CarMaker lets you create multiple scenarios for your simulation and submit them in parallel to your AWS infrastructure. This greatly reduces the time to run the experiments, improving the efficiency of your simulations.

You can navigate to the Generate Scenarios page of the Scenario Manager to select a carmaker scenario template from the dropdown, like in Figure 4. Then you can provide scenario configurations like the directory path to store your scenarios, Name Value parameters of your simulation, as well as their ranges and step values. Once the parameters are set you can click Generate Scenarios to create the test files. The test scenarios will appear in the folder path you set up.

Figure 4 – Generate simulation scenarios using CarMaker Scenario Manager

Step 3 – configure your AWS Batch jobs

In the final step before running your simulation you need to configure your AWS Batch jobs.

In the AWS Batch Job Configuration page of CarMaker Scenario Manager you can configure Batch job parameters like the queue that will be scheduling your jobs, job name, and job definition blueprint. You can rely on job queue and job definition created in Step 1, or use your own by referencing their names in the CarMaker Scenario Manager UI, as shown in Figure 5.

Each Batch job can run a single, headless, CarMaker simulation at a time – eliminating the need for user input. However, by using the Batch Multi-Containers and Array Jobs, you can scale your simulation efforts significantly. You can achieve this by increasing the array size, which determines the number of parallel job containers to execute. For example, an array size of 6 would result in six simulation jobs running simultaneously. Within each array job, 7 containers get used, with one container hosting the headless CarMaker simulator, and the remaining six representing different ADAS Features operating in a ROS2 environment. Therefore, there are in total 42 containers in this example, which allows for substantial scaling opportunities.

Finally, you need provide the Amazon S3 bucket to store your simulation results and the startup files created in Step 2. Once you set your desired variables, you can save the configuration, update the job definition, and copy your startup files to the Amazon S3 bucket.

Figure 5 – Set AWS Batch parameters for your simulation job

Step 4 – submit your simulation jobs

Now that you have defined all parameters you can go to CarMaker Scenario Manager Home page to submit your simulation. Click on Submit Jobs to run your simulation using AWS Batch.

AWS Batch will manage job submission and bin-pack your jobs efficiently on Amazon EC2 infrastructure. When your jobs finish, CarMaker will deliver results back to your Amazon S3 bucket and Batch will automatically scale down your compute infrastructure (assuming there are no more jobs waiting).

Figure 6 – Submit your CarMaker simulation jobs to the cloud

Step 5 – examine the results

You can track the status of your simulation jobs in the Batch console. The job status will change to Succeeded when your jobs successfully finish. Successful jobs will deposit their output in the Results folder of your Amazon S3 bucket as .erg files.

Now you can download the results and analyze them locally.

Figure 7 – Examine results of your simulation

Conclusion

In this post, we’ve explored running multi-container AV/ADAS simulations in the cloud using IPG CarMaker with AWS Batch. Hopefully, you’ll take away how easy this was to implement, thanks to great integration work from IPG CarMaker to take advantage of critical features in Batch.

Multi-container simulations make it easier for engineers to model complex scenarios involving systems of systems or fleets of autonomous vehicles. With this integration between IPG CarMaker and Batch, automotive engineers and DevOps can run simulations at scale, without the burden of managing compute infrastructure – all the while shrinking the time to results and paving the way to the safer and more efficient autonomous mobility. If you are interested in leveraging IPG CarMaker and Batch Multi-Containers feature to scale your AV/ADAS simulations, please contact Ryan Qi or David Howarth. You may also watch this presentation for more information and a demonstration.