How do I troubleshoot the error "unable to pull secrets or registry auth" in Amazon ECS?
Last updated: 2022-04-07
I received one of the following errors when I launched an Amazon Elastic Container Service (Amazon ECS) task:
ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried
The AWS Fargate platform version 1.4.0 uses the task elastic network interface to pull the image and secrets. All network traffic flows through the elastic network interface within your Amazon Virtual Private Cloud (Amazon VPC). You can view this traffic through your Amazon VPC Flow Logs. However, the task uses your network configuration instead of using the elastic network interfaces owned by Fargate. This is because the elastic network interfaces are placed within your Amazon VPC.
The Amazon ECS container agent uses the task execution AWS Identity and Access Management (IAM) role to get information from the AWS Systems Manager Parameter Store or AWS Secrets Manager. If you use a customer managed AWS Key Management Service (AWS KMS) key for encrypting data in the Systems Manager Parameter Store or Secrets Manager, then you must grant permissions to the task execution IAM role for the following actions: ssm:GetParameters, secretsmanager:GetSecretValue, and kms:Decrypt.
Check the routes from your subnets to the internet
If you have a Fargate task in a public subnet, then verify that your task has an assigned public IP address and a default route (0.0.0.0/0) to an internet gateway. Be sure to enable Auto-assign public when you launch a new task or create a new service. If you use the Secrets Manager or Systems Manager VPC endpoints in a public subnet, and you have AmazonProvidedDNS enabled in your Amazon VPC's DHCP options setting, then the Amazon VPC endpoint is used to reach the Secrets Manager or Systems Manager instead of the internet gateway in the public subnet.
Note: You can't enable this option for existing tasks. For existing services, you can't enable this option using the AWS Management Console. However, you can use the AWS Command Line Interface (AWS CLI) to reconfigure the existing services. If you created your Amazon ECS service using an AWS CloudFormation stack, then you can update the service by modifying the NetworkConfiguration property of AWS::ECS::Service.
If you have a Fargate task in a private subnet, then verify that your task has a default route (0.0.0.0/0) to a NAT gateway, AWS PrivateLink, or another source of internet connectivity.
- If you use a NAT gateway, then place your NAT gateway in a public subnet. For more information, see Architecture with an internet gateway and a NAT gateway.
- If you use PrivateLink, then be sure that your Fargate infrastructure can use the security groups for your Amazon VPC endpoints.
- If you use a custom name domain server, then be sure that the DNS query has outbound access on port 53 using UDP and TCP protocol and HTTPS access on port 443.
Check your network ACL and security group settings
Verify that your network access control list (ACL) and security groups don't block outbound access to port 443 from the subnet. For more information, see Security groups for your VPC.
Note: Fargate tasks must have outbound access to port 443 to allow outgoing traffic and access Amazon ECS endpoints.
Check your Amazon VPC endpoints
If you use PrivateLink, then be sure that you created the required endpoints.
The required endpoints for Fargate platform versions 1.4.0 or later are the following:
- S3 gateway endpoint
For more information, see Considerations for Amazon ECR VPC endpoints.
Note: If your task definition uses Secrets Manager, Systems Manager parameters, or Amazon CloudWatch Logs, then you might need to define endpoints. For more information, see Using Secrets Manager with VPC endpoints, Creating the VPC endpoints for Amazon ECS, and Using CloudWatch Logs with interface VPC endpoints.
If you use PrivateLink, then be sure that the security group attached to the Amazon VPC endpoint allows traffic from the Fargate task security group or Fargate task VPC CIDR range on TCP port 443.
Check the VPC endpoint policies and Endpoint policies for Amazon Simple Storage Solution (Amazon S3) to be sure that the Fargate infrastructure is allowed to access the services.
Check your IAM roles and permissions
The task execution role grants the required permissions to the Amazon ECS container and Fargate agents to make API calls for the task. This role is required by Fargate when you do the following:
- Pull a container image from Amazon Elastic Container Registry (Amazon ECR).
- Use the awslogs log driver.
- Use private registry authentication.
- Reference sensitive data using the Secrets Manager secrets or Systems Manager Parameter Store parameters.
If your use case involves any of the preceding scenarios, then be sure that you have the required permissions defined in your task execution role. For a complete list of required permissions, see Amazon ECS task execution IAM role.
Check the referenced sensitive information in the Amazon ECS task definition
Check if the secret/parameter names are matching the referenced names in your Amazon ECS task definition. Then, check if the values in the container definition in your task definition are matching the values in your Amazon ECS task definition. For more information, see How can I pass secrets or sensitive information securely to containers in an Amazon ECS task?
If the Systems Manager Parameter Store parameter exists in the same Region as the task you are launching, then you can use either the full ARN or the name of the secret. If the parameter exists in a different Region, then the full ARN must be specified.
To check the System Manager parameter name and ARN, do the following:
1. Open the AWS Systems Manager console.
2. In the navigation pane, choose Parameter Store, and confirm your Parameter store name.
3. Then, to get ARN information about the parameter, use the AWS Command Line Interface (AWS CLI), and run the following command:
Note: Replace name_of_parameter_store_secret with your Parameter store secret name.
$ aws ssm get-parameter —name <name_of_parameter_store_secret> —with-decryption
Note: Parameters that reference Secrets Manager secrets can't use the Parameter Store versioning or history features. For more information, see Restrictions.