How do I get SSM agent logs for Fargate tasks that have Amazon ECS Exec activated?

Short description

Before you start using Amazon ECS Exec, see Prerequisites for using ECS Exec.

To get SSM agent logs for Fargate tasks that have ECS Exec activated, create an Amazon Elastic File System (Amazon EFS) file system. Then, mount the Amazon EFS file system on the Fargate container. Finally, mount the same file system on an Amazon Elastic Compute Cloud (Amazon EC2) instance to get the SSM agent logs.

Important: Your Amazon EFS file system, Amazon ECS cluster, and Fargate tasks must all be in the same Amazon Virtual Private Cloud (Amazon VPC).

Note: The following resolution is only for Fargate tasks that have ECS Exec activated. Use the resolution steps solely for debugging. Launch it as a standalone task, or keep your desiredCount of tasks to "1" in your Amazon ECS service to avoid overriding logs. You can also use the following resolution for scenarios where you must check non stderr/stdout logs from the containers.

Resolution

Create your Amazon EFS file system and mount it on a Fargate container in a Task or Service.

1. Create your Amazon EFS file system.
2. Note the Amazon EFS ID and security group ID.
3. Edit your file system security group rules to allow inbound connections on port 2049 from the security group that's associated with your Fargate task.
4. Update your Amazon ECS security group to allow outbound connections on port 2049 to your file system's security group.
5. Open the Amazon ECS console, and navigate to your Amazon ECS task definition.
6. In the Volumes section, choose Add volume.
7. For Name, enter a name for your volume.
8. For Volume type, enter "EFS".
9. For File system ID, enter the ID for your file system.
10. In the Containers definition section, navigate to the STORAGE AND LOGGING section, and select the volume that you created for the source volume.
11. For Container path, select /var/log/amazon.
12. Update the Fargate service or task with the task definition that you created.

Mount the Amazon EFS file system on an Amazon EC2 instance and get the SSM Agent logs

1.    Mount your file system on an EC2 instance.

2.    Run the following command to get the log data:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-01b0bxxxxxxxx.efs.ap-southeast-1.amazonaws.com:/ /efs

Example output:

# df -h
Filesystem                                          Size  Used Avail Use% Mounted on
fs-01b0bxxxxxxxx.efs.us-west-2.amazonaws.com:/      8.0E     0  8.0E   0% /efs

The following is an example of logs stored at path /var/log/amazon/ssm/amazon-ssm-agent.log in the Fargate container:

[root@ip-172-31-32-32 efs]# cd ssm/
[root@ip-172-31-32-32 ssm]# ls
amazon-ssm-agent.log  audits
[root@ip-172-31-32-32 ssm]# cat amazon-ssm-agent.log | tail -n 10
2022-10-20 11:50:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread
2022-10-20 11:50:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started
2022-10-20 11:50:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done
2022-10-20 11:55:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started
2022-10-20 11:55:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done
2022-10-20 12:00:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] started idempotency deletion thread
2022-10-20 12:00:34 WARN [ssm-agent-worker] [MessageService] [MessageHandler] [Idempotency] encountered error open /var/lib/amazon/ssm/170b15cacf5846ed836bcd7903cbee48-2531612879/idempotency: no such file or directory while listing replies in /var/lib/amazon/ssm/170b15cacf5846ed836bcd7903cbee48-2531612879/idempotency
2022-10-20 12:00:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread
2022-10-20 12:00:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started
2022-10-20 12:00:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done
[root@ip-172-31-32-32 ssm]#