Advice for customers dealing with Docker Hub rate limits, and a Coming Soon announcement

Many container customers building applications use common software packages (e.g. operating systems, databases, and application components) that are publicly distributed as container images on Docker Hub. Docker, Inc. has announced that the Hub service will begin limiting the rate at which images are pulled under their anonymous and free plans. These limits will progressively take effect beginning November 2, 2020. Once fully in place, free plan anonymous use will be limited to 100 pulls per six hours, free plan authenticated accounts limited to 200 pulls per six hours, and Pro and Team accounts will not see any rate limits.

With the introduction of these limits, our customers should expect some of their applications and tools that use public images from Docker Hub to face throttling errors, such as when they build from a parent public image or pull a public image to run. Many of our customers have expressed concern about possible impact, so we are sharing some practical advice for managing Docker Hub’s rate limits, and announcing an upcoming AWS solution.

Managing Docker Hub’s pull rate limit impact in the short-term

If customers see impact, the overall approach we recommend is first to identify public container images in use, then choose a mitigation approach and take the necessary steps.

You can identify Docker Hub public images by searching for files named Dockerfile in source code for the FROM command. If there is no registry URL preceding the image and tag, that image will pull from Docker Hub when it runs. For example, FROM amazonlinux:latest pulls the latest amazonlinux version from Docker Hub as the parent image in an application image build. For completeness, also search your container application cluster configurations for public images. To automate the detection of public images served from Docker Hub you can use awslabs/aws-container-images-toolkit, a tool we developed that allows you to generate a list of public images in code repositories, Amazon Elastic Container Service (ECS), as well as Amazon Elastic Kubernetes Service (EKS) and self-managed Kubernetes clusters.

There are two mitigation approaches we recommend. One, copy public images being used into a private registry such as Amazon Elastic Container Registry (ECR). Or two, upgrade to a paid Docker Hub subscription. Both approaches require switching to an authenticated pull model. ECS customers can follow these instructions to authenticate task pull requests. Kubernetes customers, including EKS users can follow these instructions to authenticate worker node or pod pulls. Note that public images copied to ECR may be pulled without additional authentication effort if AWS Identity and Access Management (IAM) prerequisites are in place. IAM prerequisites for authenticated pulls are documented for ECS here and for EKS here. Customers also using third-party systems can consult this Docker link for additional authenticated pull documentation.

Amazon ECS Agent

AWS publishes the Amazon ECS agent as a public container image and pre-installs it on all ECS-optimized Amazon Machine Images (AMIs). Any instances using the ECS-optimized AMIs do not depend on Docker Hub to launch the instance, so customers using these AMIs will not be impacted by Docker Hub’s limitations on pulls of the ECS agent container image. Customers who update the agent to the latest version using the AWS Console or CLI will also not see rate limit impacts when upgrading, as the upgrade process downloads the latest ECS agent image from Amazon S3 and does not rely on Docker Hub.

Customers who create their own AMIs for use with ECS, or use third-party AMIs, must manage installation and upgrades of the ECS agent. These customers can store their own copy of the ECS agent in a private registry (such as ECR) and pull it from there, download it from a public S3 bucket owned by AWS, or pull the image from Docker Hub. Customers downloading the ECS agent from Docker Hub may be subject to Docker Hub rate limits. To mitigate rate limit effects, these customers can take the steps outlined above to manage Docker’s pull rate impact.

Amazon EKS Add Ons

All EKS add-on software that is included with EKS clusters is hosted on ECR today and will not be subject to Docker Hub rate limits. EKS-built and maintained projects, such as the AWS Load Balancer Controller include helm charts that reference images on ECR as well. However, manifests for these projects available on GitHub include references to images that are published to Docker Hub. You can check whether you are using a manifest with a Docker Hub image using the steps described in this blog and update to use a chart that references an ECR hosted image. These charts are available in the EKS Charts GitHub repository.

Coming soon, a reliable solution for public container images

Today, Amazon Elastic Container Registry (ECR) customers download billions of images each week, enabling them to deploy containers for use in their own applications running on-premises and in AWS. Events like Docker Hub rate limits reinforce the reality that public container images are a critical part of the container software supply chain. Customers have asked us to help make public container images they use as highly available and dependable as their images hosted in ECR.

We have been working for many months to answer this customer ask. Within weeks, AWS will deliver a new public container registry that will allow developers to share and deploy container images publicly. This new registry will allow developers to store, manage, share, and deploy container images for anyone to discover and download. Developers will be able to use AWS to host both their private and public container images, eliminating the need to use different public websites and registries. Public images will be geo-replicated for reliable availability around the world and offer fast downloads to quickly serve up images on-demand. Anyone (with or without an AWS account) will be able to browse and pull containerized software for use in their own applications. Developers will be able use the new registry to distribute public container images and related files like helm charts and policy configurations for use by any developer. A new website will allow anyone to browse and search for public container images, view developer provided details, and see pull commands — all without needing to sign in to AWS. AWS-provided public images such as the ECS agent, Amazon CloudWatch agent, and AWS Deep Learning Container images will also be available.

AWS customers will enjoy a smooth, dependable experience when working with public container images, and with pricing that provides additional value. A developer sharing public images on AWS will get 50 GB of free storage each month and will pay nominal charges after. Anyone who pulls images anonymously will get 500 GB of free data bandwidth each month after which they can sign up or sign in to an AWS account. Simply authenticating with an AWS account increases free data bandwidth up to 5 TB each month when pulling images from the internet. And finally, workloads running in AWS will get unlimited data bandwidth from any region when pulling publicly shared images hosted on AWS.