How do I troubleshoot Amazon ECR issues with Amazon EKS?

Last updated: 2020-01-27

I can't pull images from Amazon Elastic Container Registry (Amazon ECR) when I use Amazon Elastic Kubernetes Service (Amazon EKS).

Short Description

You can't pull images from Amazon ECR for one of the following reasons:

  • You can't communicate with Amazon ECR endpoints.
  • You don't have the appropriate permissions in the instance profile attached to your worker node to pull images from a particular Amazon ECR repository.

To resolve these issues, complete the steps in one of the following sections:

  • Troubleshoot communication between worker nodes and Amazon ECR endpoints
  • Update the instance profile of your worker nodes and Confirm that your repository policies are correct

Resolution

Troubleshoot communication between worker nodes and Amazon ECR endpoints

If your worker nodes can't communicate with the Amazon ECR endpoints, you could receive the following error message:

Failed to pull image "ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag": rpc error: code = Unknown desc = 
Error response from daemon: Get https://ACCOUNT.dkr.ecr.REGION.amazonaws.com/v2/: net/http: 
request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

To resolve this error, confirm the following:

  • The subnet for your worker node has a route to the internet. Check the route table associated with your subnet.
  • The security group associated with your worker node allows outbound internet traffic.
  • The ingress and egress rule for your network access control lists (ACLs) allows access to the internet.

Update the instance profile of your worker nodes

If your worker node's instance profile doesn't have permission to pull images from Amazon ECR, then you could receive the following error message from your Amazon EKS pod:

Warning  Failed     14s (x2 over 28s)  kubelet, ip-000-000-000-000.us-west-2.compute.internalFailed to pull image "ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag": rpc error: code = Unknown desc = Error response from daemon: Get https://ACCOUNT.dkr.ecr.REGION.amazonaws.com/v2/imagename/manifests/tag: no basic auth credentials
Warning  Failed     14s (x2 over 28s)  kubelet, ip-000-000-000-000.us-west-2.compute.internal  Error: ErrImagePull
Normal   BackOff    2s (x2 over 28s)   kubelet, ip-000-000-000-000.us-west-2.compute.internal  Back-off pulling image "ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag"
Warning  Failed     2s (x2 over 28s)   kubelet, ip-000-000-000-000.us-west-2.compute.internal  Error: ImagePullBackOff

To resolve this error, confirm that your worker nodes use the AmazonEC2ContainerRegistryReadOnly AWS Identity and Access Management (IAM) managed policy. Or, update the Amazon Elastic Compute Cloud (Amazon EC2) instance profile of your worker nodes with the following IAM permissions:

"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:GetLifecyclePolicy",
"ecr:GetLifecyclePolicyPreview",
"ecr:ListTagsForResource",
"ecr:DescribeImageScanFindings"

Important: It's a best practice to use the AmazonEC2ContainerRegistryReadOnly policy instead of creating a duplicate policy.

The updated instance profile gives your worker nodes the permissions to access Amazon ECR and pull images through the kubelet. The kubelet is responsible for fetching and periodically refreshing Amazon ECR credentials. For more information, see Kubernetes Images.

Confirm that your repository policies are correct

Repository policies are a subset of IAM policies that control access to individual Amazon ECR repositories. IAM policies are generally used to apply permissions for the entire Amazon ECR service, but can also control access to specific resources.

1.    Open the Amazon ECR console for your primary account.

2.    On the navigation pane, choose Repositories, and then choose the repository that you want to check.

3.    On the navigation pane, choose Permissions, and then check if your repository has the correct permissions.

The following example policy allows a specific IAM user to describe the repository and the images within the repository:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "ECR Repository Policy",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/MyUsername"
      },
      "Action": [
        "ecr:DescribeImages",
        "ecr:DescribeRepositories"
      ]
    }
  ]
}

Did this article help you?

Anything we could improve?


Need more help?