How do I troubleshoot my container logs for Amazon ECS or Amazon EKS?

Last updated: 2019-07-15

I can't find the container logs for my Amazon Elastic Container Service (Amazon ECS) task or Amazon Elastic Kubernetes Service (Amazon EKS) pod. Where can I find these logs?

Short Description

Your containerized application might not write some or all of the logs when you:

  • Run the "docker logs yourContainerName" command on a container instance in Amazon ECS.
  • Use the awslogs log driver for a task in Amazon ECS.
  • Run the "kubectl logs yourPodName" command for an Amazon EKS cluster.

You can experience these issues when:

  • There are problems with the host instance.
  • Your logs are not visible to the Docker daemon because your containerized application isn't writing to the correct location.

Resolution

Find the logs for your Amazon ECS tasks

To confirm that your task is configured correctly so that you can find your logs, consider the following:

  • The log driver is set by the logConfiguration parameter in the container definition section of your ECS task definition. Log drivers are set per container. If your ECS task has multiple container definitions, then verify that the log configuration is correct for the containers with your logs.
  • The Amazon ECS container agent automatically removes log files to reclaim free space. To preserve your log files for longer on your container instance, you can reduce the frequency of your task cleanup.

If your tasks are using the awslogs log driver, then the following conditions are true:

  • Logs are streamed to Amazon CloudWatch Logs. These logs are never written to the container instance.
  • The "docker logs yourContainerName" command returns the following error message: "Error response from daemon: configured logging driver does not support reading."

To allow EC2 launch types to stream to CloudWatch Logs, grant permissions on the IAM role used by your container instance.

To allow Fargate launch types to stream to CloudWatch Logs, grant permissions on the IAM role used by your task.

If your logs are still missing, complete the steps in the Troubleshoot the container section.

Find the logs for your Kubernetes pods on Amazon EKS

To return the log files generated from a pod's container, run the following kubectl command:

kubectl logs yourPodName

Note: The kubelet automatically removes log files after a pod exits, as part of garbage collection. To preserve these log files for longer on a worker node, configure the kubelet to run garbage collection less frequently.

If your logs are still missing, complete the steps in the Troubleshoot the container section.

Troubleshoot the container

Consider the following options for your application:

  • Set your application to the correct log level during your container build.
    Note: Depending on your application, you might need to set logging through an environment variable or in a configuration file.
  • Make your application the ENTRYPOINT of the container.
    Note: The ENTRYPOINT in your dockerfile is the process where STDOUT and STDERR I/O streams are logged.
  • Build your container with application log files linked to STDOUT and STDERR, or configure your application to log directly to /proc/1/fd/1 (stdout) and /proc/1/fd/2 (stderr). For examples, see the official nginx and httpd container images.
    Note: If possible, make your application process the parent process in your container. If your container uses a shell script as the ENTRYPOINT, configure the script to initialize your container data at runtime.

Did this article help you?

Anything we could improve?


Need more help?