How can I update my private repository credentials on an Amazon ECS container agent?

Short description

You can supply private repository credentials to the Amazon ECS container agent using either Secrets Manager in your task definition or environment variables.

Choose either of the following options:

• If you're using Secrets Manager to supply private repository credentials, then follow the steps in the Update your private repository credentials with Secrets Manager section.
• If you're using instance user data and environment variables to supply private repository credentials, then follow the steps in the Update your private repository credentials with environment variablessection.

Note: It's not a security best practice to store credentials in plaintext on container instances. It's a best practice to use Secrets Manager instead, unless there's a direct architectural dependency on storing registry credentials within the container instance.

Resolution

Update your private repository credentials with Secrets Manager

1.    Open the Secrets Manager console.

2.    Choose your secret, and then choose Retrieve secret value.

3.    Choose Edit.

4.    Update the stored credentials for your private registry, and then choose Save.

To continue, follow the steps in the Test your updated private repository credentials section.

Update your private repository credentials with environment variables

1.    Connect to your container instance.

2.    To find out how you're supplying Docker credentials to your ECS container agent, run the following command:

$ cat /etc/ecs/ecs.config

This command returns the contents of the /etc/ecs/ecs.configfile.

If the ECS_ENGINE_AUTH_TYPE variable is set to docker, then you're directly passing your Docker credentials by plaintext to your ECS container agent. You should avoid this approach. Instead, use Secrets Manager, or the dockercfg format approach in the Get a new Docker authentication value section.

If the ECS_ENGINE_AUTH_TYPE variable is set to dockercfg, then you're passing your Docker credentials by a Docker-generated authentication value generated by the docker login command. To continue this approach, complete the steps in the Get a new Docker authentication value section.

Get a new Docker authentication value

1.    To log in to Docker locally, run the following command, and then enter your new credentials:

$ docker login

2.    To concatenate your config.json file, run the following command, and then copy the Docker-generated authentication key value:

$ cat ~/.docker/config.json

3.    To update the ECS_ENGINE_AUTH_DATA variable, run the following command:

$ sudo vi /etc/ecs/ecs.config

4.    In the vi editor, update the value of the ECS_ENGINE_AUTH_DATA variable to the Docker authentication key value from step 2. For example:

ECS_CLUSTER=TestECSCluster
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth","a2vpdGhhd3M6UGFzc3dvcmQ="}}

To continue, follow the steps in the Restart your ECS container agent section.

Restart your ECS container agent

1.    To restart your ECS container agent, run either of the following commands based on the Amazon Machine Images (AMIs) that your container instances are running on.

Amazon Linux ECS-optimized AMIs:

$ sudo stop ecs && sudo start ecs

Amazon Linux 2 ECS-optimized AMIs:

$ sudo systemctl restart ecs

Note: To update the /etc/ecs/ecs.config file on other container instances in your ECS cluster, go back to the Update your private repository credentials with environment variablessection. Then, follow the update process for each container instance.

To continue, follow the steps in the Test your updated private repository credentials section.

Test your updated private repository credentials

The following steps assume that you're deploying an updated image across your cluster.

1.    Open the Amazon ECS console.

2.    In the navigation pane, choose Clusters, and then select your cluster.

3.    Select your service, then choose Update.

4.    Select the Force new deployment check box.

5.    For Minimum healthy percent, enter 50.

6.    Complete the remaining steps in the setup wizard, and then choose Update Service.

7.    Choose View Service.

8.    On the Deployments tab, view the new deployment. Amazon ECS gradually stops tasks under the previous deployment, and then restarts the tasks under the new deployment while attempting a fresh image pull.

Note: Step 8 assumes that your cluster has enough resources to successfully perform a rolling update deployment type.

9.    Choose the Tasks tab, and then check each individual task and its status.

If the task status is set to Running, then the service updated this task successfully without error.

If the task status is set to Running (CannotPullContainerError), then the service updated this task, but there's an error. The ECS container agent can't pull a new container image and is using the old cached image. Verify that your credentials were updated, and then perform another service deployment update.

Note: To display the full details of your task and see the "pull access denied" error, choose the dropdown arrow for your individual task status. For example:

CannotPullContainerError: Error response from daemon: pull access denied for user/reponame, repository does not exist or may require 'docker login'

---

Related information

Amazon ECS container agent configuration

Authentication formats