AWS Deep Learning Containers for PyTorch 2.4 Inference on EC2, ECS and EKS

AWS Deep Learning Containers (DLCs) for Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), and Amazon Elastic Kubernetes Service (EKS) are now available with PyTorch 2.4 and support for CUDA 12.4 on Ubuntu 22.04. You can launch the new versions of the Deep Learning Containers on any of the EC2, ECS and EKS services. For a complete list of frameworks and versions supported by the AWS Deep Learning Containers, see below.

This release includes container images for inference on CPU and GPU, optimized for performance and scale on AWS. These Docker images have been tested with EC2, ECS and EKS services, and provide stable versions of NVIDIA CUDA, cuDNN, Intel MKL, and other components to provide an optimized user experience for running deep learning workloads on AWS. All software components in these images are scanned for security vulnerabilities and updated or patched in accordance with AWS Security best practices. These new DLC are designed to be used on any of the EC2, ECS and EKS services. If you are looking for a DLC to use with SageMaker, please refer to this documentation.

A list of available containers can be found in our documentation. For latest updates, please also see the aws/deep-learning-containers GitHub repo. Get started quickly with the AWS Deep Learning Containers using the getting-started guides and beginner to advanced level tutorials in our developer guide. You can also subscribe to our discussion forum to get launch announcements and post your questions.

Release Notes

• Introduced containers for PyTorch 2.4.0 for inference supporting EC2, ECS, and EKS. For details about this release, check out our GitHub release tag.
• PyTorch 2.4 offers support for python custom operator API allowing users to integrate custom kernels such as Triton kernels into torch.compile.
• Please refer to the official PyTorch 2.4 release notes here for the full description of updates.
• Added Python 3.11 support
• Added CUDA 12.4 support
• Added Ubuntu 22.04 support
• The Dockerfile for CPU can be found here, and the Dockerfile for GPU can be found here.

Security Advisory

AWS recommends that customers monitor critical security updates in the AWS Security Bulletin.

Python 3.11 Support

Python 3.11 is supported in the PyTorch Inference containers.

CPU Instance Type Support

The containers support x86_64 CPU instance types.

GPU Instance Type support

The containers support GPU instance types and contain the following software components for GPU support:

• CUDA 12.4.1
• cuDNN 9.1.0.70+cuda12.4
• NCCL 2.22.3+cuda12.4

AWS Regions support

The containers are available in the following regions:

Region	Code
US East (Ohio)	us-east-2
US East (N. Virginia)	us-east-1
US West (Oregon)	us-west-2
US West (N. California)	us-west-1
AF South (Cape Town)	af-south-1
Asia Pacific (Hong Kong)	ap-east-1
Asia Pacific (Hyderabad)	ap-south-2
Asia Pacific (Mumbai)	ap-south-1
Asia Pacific (Osaka)	ap-northeast-3
Asia Pacific (Seoul)	ap-northeast-2
Asia Pacific (Tokyo)	ap-northeast-1
Asia Pacific (Melbourne)	ap-southeast-4
Asia Pacific (Jakarta)	ap-southeast-3
Asia Pacific (Sydney)	ap-southeast-2
Asia Pacific (Singapore)	ap-southeast-1
Central (Canada)	ca-central-1
EU (Zurich)	eu-central-2
EU (Frankfurt)	eu-central-1
EU (Ireland)	eu-west-1
EU (London)	eu-west-2
EU( Paris)	eu-west-3
EU (Spain)	eu-south-2
EU (Milan)	eu-south-1
EU (Stockholm)	eu-north-1
Middle East (Bahrain)	me-south-1
Middle East (UAE)	me-central-1
SA (Sau Paulo)	sa-east-1
China (Beijing)	cn-north-1
China (Ningxia)	cn-northwest-1

Build and Test

• Built on: c5.18xlarge
• Tested on: c5.18xlarge, g3.16xlarge, m5.16xlarge, t3.2xlarge, p3.16xlarge, p3dn.24xlarge, p4d.24xlarge, g4dn.xlarge
• Tested with MNIST and Resnet50/ImageNet datasets on EC2, ECS AMI (Amazon Linux AMI 2.0.20221102), and EKS AMI (amazon-eks-gpu-node-1.25.16-20240307)