How do I troubleshoot issues related to AWS Secrets Manager secrets in Amazon ECS?

Last updated: 2022-03-30

When I try to place a task that uses AWS Secrets Manager secrets in Amazon Elastic Container Services (Amazon ECS), I get one of the following errors:

  • ResourceInitializationError error on AWS Fargate
  • AccessDenied error on Amazon Elastic Compute Cloud (Amazon EC2)

The task is unable to retrieve secrets from Secrets Manager and fails.

Short description

You get these error messages due to one of the following reasons:

  • The Amazon ECS task execution role doesn't have the required permissions to access Secrets Manager secrets.
  • Secrets Manager can't find the specified secret.
  • The name of the secret ends with a hyphen followed by six characters. This returns unexpected results when you search for a secret using partial ARN.
  • There are issues with your Amazon Virtual Private Cloud (Amazon VPC) networking configuration.

Resolution

Check that the ECS task execution role has the required permissions

  1. Open the IAM console.
  2. In the navigation pane, choose Roles.
  3. Search the list of roles for the task execution role that your ECS tasks are using.
  4. Confirm that the task execution role has the required permissions to access the necessary Secrets Manager resource.

Check that the Secrets Manager secret exists

  1. Open the Secrets Manager console.
  2. On the Secrets list page, choose the secret that your ECS tasks are using.
  3. Confirm that your ECS task definition includes these specific secret names. If you specified additional parameters in the secret, confirm that your task definition includes json-key/version-stage/version-id that's specified in the secret.

Check the name and ARN of the Secrets Manager secret

Check if your secret name ends with a hyphen followed by six characters (example: myappsecret-xxxxxx). Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN. Therefore, it's a best practice not to end your secret name with a hyphen followed by six characters. Doing so might return unexpected results when you search for a secret using partial ARN. To resolve this issue, be sure that you use the full ARNs, including the additional hyphen and six characters, of the secrets from Secrets Manager.

Verify that your VPC networking configuration allows your Amazon ECS infrastructure to reach Secrets Manager

Create the interface VPC endpoints for Secrets Manager when the following conditions are true:

  • You're using a task definition that references the Secrets Manager secrets to retrieve sensitive data for your containers.
  • You're using interface VPC endpoints.

Also, be sure that the security groups for your VPC endpoints allow the ECS infrastructure to use these endpoints.

To check if the VPC endpoint for Secrets Manager exists, do the following:

  1. Open the Amazon VPC console.
  2. In the navigation pane, choose Endpoints.
  3. Be sure that the VPC endpoint com.amazonaws.example-region.secretsmanager for Secrets Manager is on the list of endpoints.
  4. Select the endpoint from the list of endpoints, and then choose the Subnets tab. Be sure that the subnets for this endpoint include the ones used by your ECS task.
    Note: If no subnets are listed, then choose Manage Subnets. Select the subnet based on its Availability Zone, and then choose Modify Subnets.

To confirm that the security group attached to the endpoint com.amazonaws.example-region.secretsmanager allows incoming connections on port 443 from the Amazon ECS tasks, do the following:

  1. Select the endpoint from the list of endpoints.
  2. Choose the Security Groups tab.
  3. Choose the Group ID for the security group that you want to check.
  4. Choose the Inbound rules tab.
  5. Verify that the list of inbound rules includes a rule that allows 443 connections from your ECS tasks.