How do I resolve the error "CannotPullContainerError: You have reached your pull rate limit" in Amazon ECS?

Last updated: 2022-03-25

When I try to pull images from Docker Hub, my Amazon Elastic Container Service (Amazon ECS) task fails with the following error:

CannotPullContainerError: inspect image has been retried 5 time(s): httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/manifests/sha256:2bb501e6429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Short description

You get this error when you try to pull an image from the public Docker Hub repository after you have reached your Docker pull rate limit, and a http status code of 429 is returned. Docker Hub uses IP addresses to authenticate the users, and pull rates limits are based on individual IP addresses. For anonymous users, the rate limit is set to 100 pulls per 6 hours per IP address. For authenticated users with a Docker ID, the pull rate is set to 200 pulls per 6-hour period. If your image pull request exceeds these limits, these requests are denied until the six-hour window elapses. If you're running your Amazon ECS/Amazon EKS workload, then most of the data is pulled through a NAT Gateway that has a fixed IP address. In this case, the request gets throttled when you exceed the pull limit.

Resolution

Use one of the following workarounds to troubleshoot this issue.

Copy public images into an Amazon ECR private registry

Create an Amazon Elastic Container Registry (Amazon ECR) repository, and then push the image into this new repository. With this approach, you might avoid exceeding the Docker Hub pull limit by pulling the images from the Amazon ECR repository.

1.    Run a command similar to the following to pull the image from Docker Hub:

docker pull example-image

2.    Run a command similar to the following to authenticate your Docker client to access the Amazon ECR registry:

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com

3.    Run a command similar to the following to tag the image to push to your repository:

docker tag myrepository:latest 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com/myrepository:latest

4.    Run a command similar to the following to push the Docker image to an Amazon ECR registry:

docker push 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com/myrepository:latest

5.    Run a command similar to the following to update the Docker file to use the newly pushed Amazon ECR image as the base image:

FROM 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com/myrepository:tag

Be sure to replace the following in the preceding commands:

  • example-image with the name of the public image that you want to push
  • 1111222233334444 with your account ID
  • myrepository:latest with the name of your Amazon ECR registry
  • eu-west-1 with the Region of your choice

Authenticate the Docker Hub pull

If you authenticate with Docker Hub, you have more rate limits as an authenticated user and are rate-limited based on the Docker username. You can store your Docker Hub username and password as a secret in AWS Secrets Manager, and then use this secret to authenticate to Docker Hub.

Create a Secrets Manager secret for Docker Hub credentials

To create a secret for your Docker Hub credentials, use the instructions under the section To create a basic secret in Enabling private registry authentication.

Update your task execution AWS Identity and Access Management (AWS IAM) role

To grant access for the Amazon ECS task to the secret that you created, you must manually add the required permissions as an inline policy to the task execution role.

1.    Open the IAM console.

2.    In the navigation pane, choose Roles.

3.    Search the list of roles for ecsTaskExecutionRole, and choose the role to view the attached policies.

4.    On the Permissions tab, choose Add permissions, and then choose Create inline policy.

5.    In the Create policy page, choose JSON, and then copy and paste the following policy:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"secretsmanager:GetSecretValue",
			"kms:Decrypt"
		],
		"Resource": [
			"arn:aws:secretsmanager:eu-west-1:1111222233334444:secret:dockerhub-0knT",
			"arn:aws:kms:eu-west-1:1111222233334444:key/mykey"
		]
	}]
}

Be sure to replace the following in the policy:

  • 1111222233334444 with your account ID
  • eu-west-1 with the Region of your choice
  • mykey with your AWS KMS key

Note: Include kms:Decrypt only if your key doesn't use the default key and uses a custom AWS Key Management Service (AWS KMS) key. Add the ARN for your custom key as a resource.

6.    Choose Review policy.

7.    For Name, enter the name of the policy (ECSSecrets).

8.    Choose Create policy.

Create a task definition that uses the secret for Docker authentication

Use the instructions in Creating a task definition using the classic console to create your Amazon ECS task definition. For Task execution role, be sure to select the task execution IAM role that you updated in the preceding section.

In the Container definitions section, do the following:

  1. Choose Add container.
  2. For Container name, enter the name of your container.
  3. For Image, enter the name of the image, or include the path to your private image (example: repository-url/image.tag).
  4. Select Private repository authentication.
  5. For Secrets Manager ARN or name, enter the ARN of the secret that you created.
  6. Choose Add.

Create an Amazon ECS cluster and run the Amazon ECS task

Create an Amazon ECS cluster. Then, run the task using the task definition that you created in the preceding section.

Use Amazon ECR public registry for public container images

Identify the public images that you're using in the Docker file. Search for these images on the Amazon ECR public gallery using appropriate search filters. You don't need to authenticate to browse the public repositories and pull images. The Amazon ECR public registry contains popular base images, including operating systems, AWS-published images, Kubernetes add-ons, and artifacts. You can avoid reaching the Docker Hub's rate limit by pulling images from the Amazon ECR public registry.

You can use these images as the source for the container image in your task definition:

ContainerDefinitions: [
 {
 ...
 Image: 'public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest'
 ...
 }
 ]

You can also choose to use these images as the base image in your Docker file:

Docker File
FROM public.ecr.aws/amazonlinux/amazonlinux:latest

Upgrade to a Docker Pro or Team subscription

You can upgrade your plan to a Docker Pro or Team subscription that offers 50,000 pulls in a 24-hour period. For more information on pricing plans, see Docker Hub pricing.


Did this article help?


Do you need billing or technical support?