Leveraging Elastic Fabric Adapter to run HPC and ML Workloads on AWS Batch
Leveraging Elastic Fabric Adapter to run HPC and ML Workloads on AWS Batch
This post is contributed by Sean Smith, Software Development Engineer II, AWS ParallelCluster & Arya Hezarkhani, Software Development Engineer II, AWS Batch and HPC
On August 2, 2019, AWS Batch announced support for Elastic Fabric Adapter (EFA). This enables you to run highly performant, distributed high performance computing (HPC) and machine learning (ML) workloads by using AWS Batch’s managed resource provisioning and job scheduling.
EFA is a network interface for Amazon EC2 instances that enables you to run applications requiring high levels of inter-node communications at scale on AWS. Its custom-built operating system bypasses the hardware interface and enhances the performance of inter-instance communications, which is critical to scaling these applications. With EFA, HPC applications using the Message Passing Interface (MPI) and ML applications using NVIDIA Collective Communications Library (NCCL) can scale to thousands of cores or GPUs. As a result, you get the application performance of on-premises HPC clusters with the on-demand elasticity and ﬂexibility of the AWS Cloud.
AWS Batch is a cloud-native batch scheduler that manages instance provisioning and job scheduling. AWS Batch automatically provisions instances according to job speciﬁcations, with the appropriate placement group, networking conﬁgurations, and any user-speciﬁed ﬁle system. It also automatically sets up the EFA interconnect to the instances it launches, which you specify through a single launch template parameter.
In this post, we walk through the setup of EFA on AWS Batch and run the NAS Parallel Benchmark (NPB), a benchmark suite that evaluates the performance of parallel supercomputers, using the open source implementation of MPI, OpenMPI.
This walk-through assumes:
- You have an AWS account.
- You are familiar with the AWS Command Line Interface (AWS CLI).
Conﬁguring your compute environment
First, configure your compute environment to launch instances with the EFA device.
Creating an EC2 placement group
The ﬁrst step is to create a cluster placement group. This is a logical grouping of instances within a single Availability Zone. The chief beneﬁt of a cluster placement group is non-blocking, non-oversubscribed, fully bi-sectional network connectivity. Use a Region that supports EFA—currently, that is us-east-1, us-east-2, us-west-2, and eu-west-1. Run the following command:
Creating an EC2 launch template
Next, create a launch template that contains a user-data script to install EFA libraries onto the instance. Launch templates enable you to store launch parameters so that you do not have to specify them every time you launch an instance. This will be the launch template used by AWS Batch to scale the necessary compute resources in your AWS Batch Compute Environment.
First, encode the user data into base64-encoding. This example uses the CLI utility base64 to do so.
$ echo "MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==MYBOUNDARY==" --==MYBOUNDARY== Content-Type: text/cloud-boothook; charset="us-ascii" cloud-init-per once yum_wget yum install -y wget cloud-init-per once wget_efa wget -q --timeout=20 https://s3-us-west-2.amazonaws.com/aws-efa-installer/aws-efa-installer-latest.tar.gz -O /tmp/aws-efa-installer-latest.tar.gz cloud-init-per once tar_efa tar -xf /tmp/aws-efa-installer-latest.tar.gz -C /tmp pushd /tmp/aws-efa-installer cloud-init-per once install_efa ./efa_installer.sh -y pop /tmp/aws-efa-installer cloud-init-per once efa_info /opt/amazon/efa/bin/fi_info -p efa --==MYBOUNDARY==--" | base64
Save the base64-encoded output, because you need it to create the launch template.
Next, make sure that your default security group is conﬁgured correctly. On the EC2 console, select the default security group associated with your default VPC and edit the inbound rules to allow SSH and All traﬃc to itself. This must be set explicitly to the security group ID for EFA to work, as seen in the following screenshot.
Then edit the outbound rules and add a rule that allows all inbound traﬃc from the security group itself, as seen in the following screenshot. This is a requirement for EFA to work.
Now, create an ecsInstanceRole, the Amazon ECS instance profile that will be applied to Amazon EC2 instances in a Compute Environment. To create a role, follow these steps.
- Choose Roles, then Create Role.
- Select AWS Service, then EC2.
- Choose Permissions.
- Attach the managed policy AmazonEC2ContainerServiceforEC2Role.
- Name the role ecsInstanceRole.
You will create the launch template using the ID of the security group, the ID of a subnet in your default VPC, and the ecsInstanceRole that you created.
Next, choose an instance type that supports EFA, that’s denoted by the n in the instance name. This example uses c5n.18xlarge instances.
You also need an Amazon Machine Image (AMI) ID. This example uses the latest ECS-optimized AMI based on Amazon Linux 2. Grab the AMI ID that corresponds to the Region that you are using.
This example uses UserData to install EFA. This adds 1.5 minutes of bootstrap time to the instance launch. In production workloads, bake the EFA installation into the AMI to avoid this additional bootstrap delay.
Now create a ﬁle called launch_template.json with the following content, making sure to substitute the account ID, security group, subnet ID, AMI ID, and key name.
Create a launch template from that ﬁle:
Creating a compute environment
Next, create an AWS Batch Compute Environment. This uses the information from the launch template
EFA-Batch-Launch-Template created earlier.
Now, create the compute environment:
Building the container image
To build the container, clone the repository that contains the Dockerﬁle used in this example.
First, install git:
In that repository, there are several ﬁles, one of which is the following Dockerfile.
To build this Dockerﬁle, run the included Makerfile with:
Now, push the created container image to Amazon Elastic Container Registry (ECR), so you can use it in your AWS Batch JobDeﬁnition:
From the AWS CLI, create an ECR repository, we’ll call it aws-batch-efa:
Edit the top of the makeﬁle and add your AWS account number and AWS Region.
To push the image to the ECR repository, run:
Run the application
To run the application using AWS Batch multi-node parallel jobs, follow these steps.
Setting up the AWS Batch multi-node job definition
Set up the AWS Batch multi-node job definition and expose the EFA device to the container by following these steps.
First, create a ﬁle called job_definition.json with the following contents. This ﬁle holds the conﬁgurations for the AWS Batch JobDeﬁnition. Speciﬁcally, this JobDeﬁnition uses the newly supported ﬁeld LinuxParameters.Devices to expose a particular device—in this case, the EFA device path /dev/infiniband/uverbs0—to the container. Be sure to substitute the image URI with the one you pushed to ECR in the previous step. This is used to start the container.
Now register the job definition
Run the job
Next, create a job queue. This job queue points at the compute environment created before. When jobs are submitted to it, they queue until instances are available to run them.
Now that you’ve created all the resources, submit the job. The numNodes=8 parameter tells the job deﬁnition to use eight nodes.
NPB is a small set of benchmarks derived from computational fluid dynamics (CFD) applications. They consist of five kernels and three pseudo-applications. This example runs the 3D Fast Fourier Transform (FFT) benchmark, as it tests all-to-all communication. For this run, use c5n.18xlarge, as conﬁgured in the AWS compute environment earlier. This is an excellent choice for this workload as it has an Intel Skylake processor (72 hyperthreaded cores) and 100 GB Enhanced Networking (ENA), which you can take advantage of with EFA.
This test runs the FT “C” Benchmark with eight nodes * 72 vcpus = 576 vcpus.
In this post, we covered how to run MPI Batch jobs with an EFA-enabled elastic network interface using AWS Batch multi-node parallel jobs and an EC2 launch template. We used a launch template to conﬁgure the AWS Batch compute environment to launch an instance with the EFA device installed. We showed you how to expose the EFA device to the container. You also learned how to package an MPI benchmarking application, the NPB, as a Docker container, and how to run the application as an AWS Batch multi-node parallel job.
We hope you found the information in this post helpful and encouraging as to all the possibilities for HPC on AWS.