Containers
Troubleshooting with Amazon ECS Exec and Amazon CloudWatch Logs Live Tail in the AWS Management Console
Container troubleshooting often needs both real-time visibility into logs and direct access to containers. Previously, these activities necessitated navigating between different AWS Management Console interfaces, setting up client connections, or learning AWS Command Line (AWS CLI) commands parameters. This context-switching increased the time to diagnose and resolve issues, particularly in production environments where container access might be restricted by network configurations.
The integration of Amazon CloudWatch Logs live tail and the Amazon ECS Elastic Container Service (Amazon ECS) Exec direct access in the Amazon ECS console centralizes container debugging capabilities. This removes the need to switch between multiple interfaces or maintain separate CLI configurations. The Amazon ECS console now incorporates essential troubleshooting capabilities that allow DevOps engineers and developers to maintain reliable container-based applications while preserving necessary security and governance controls. In this post, we explore these new features, how they work, and how to implement them in your Amazon ECS environments.
Solution overview
CloudWatch Logs Live Tail Integration provides real-time log streaming directly within the Amazon ECS console. You can use this feature to view container logs as they’re ingested, without the delay typically associated with log aggregation systems. You can integrate the filtering and pattern matching capabilities of CloudWatch Logs Live Tail directly into the Amazon ECS interface to quickly isolate and analyze specific log events while maintaining context about your container services.
Furthermore, Amazon ECS Exec with AWS CloudShell Integration enables you to run interactive commands or obtain a shell inside your running containers directly from the Amazon ECS console. This feature uses CloudShell to establish secure connections to your containers, even those running in private subnets, without needing VPN connections or bastion hosts. With just a few clicks, you can securely access your containers to perform maintenance tasks, run diagnostic commands, or execute one-off operations.
These features provide integrated troubleshooting tools for Amazon ECS users. Whether you need to monitor application behavior through logs in real-time or directly interact with containers to diagnose issues, you can now accomplish these tasks without leaving the Amazon ECS console or use CLI commands.
CloudWatch Logs Live Tail and ECS Exec with CloudShell are particularly effective in common troubleshooting scenarios. For example, when troubleshooting application behavior issues, engineers can simultaneously monitor application logs in real-time through CloudWatch Logs Live Tail while executing diagnostic commands inside the container using ECS Exec, all without leaving the Amazon ECS console. Another practical application is during deployment verification, where operations teams can observe startup sequences through CloudWatch Logs Live Tail and verify the runtime environment configuration by accessing the container shell through ECS Exec. This enables quick verification of container state and configuration. These integrations eliminate context-switching between different tools and interfaces, thus reducing debugging time.
Prerequisites
To use ECS Exec and CloudWatch Logs Live Tail in the Console, the following prerequisites are necessary:
- Amazon ECS tasks need to have ECS Exec enabled. For more information, see Monitor Amazon ECS containers with ECS Exec.
- The ECS Exec prerequisites need to be considered as listed in the Considerations section.
- You must use a task role with the appropriate permissions for ECS Exec. For more information, go to ECS Exec permissions.
- The following permissions are necessary for the AWS Identity and Access Management (IAM) user to start and stop CloudWatch Logs Live Tail sessions:
logs:DescribeLogGroupslogs:StartLiveTaillogs:StopLiveTail
- Your task definition needs to be configured to use the awslogs logging driver. For more information, go to Send logs to CloudWatch.
Walkthrough: troubleshooting database connection issues
In this section we examine a scenario where a developer deploys a Python web application with an incorrect database endpoint. The task has the database environment variable set in the task definition, but the application is unable to connect to the database, as shown in the following figure. This represents a common troubleshooting challenge that the new Amazon ECS console integrations help solve efficiently.
Figure 1: Task definition database connection string environment variable
To begin our investigation, we can use ECS Live Tail to observe the latest logs being generated by the application in real time, as shown in the following figure. From the task overview page, navigate to the Logs tab and chooe CloudWatch Logs Live Tail. When the Live Tail window opens, choose the Start button to begin monitoring the logs as they stream in. The logs immediately reveal connection failures, indicating the application cannot reach the database.
Figure 2: CloudWatch Logs Live Tail in the Amazon ECS console
Previously, after identifying these connection failures, troubleshooting would have necessitated setting up ECS exec access on a local machine or Amazon Elastic Compute Cloud (Amazon EC2) instance just to inspect the container’s runtime environment. This process created significant friction and delayed problem resolution, making it difficult to quickly verify whether the database connection string was correctly passed to the application and accessible within the container.
Now, ECS Exec with CloudShell provides direct access for confirming the application’s runtime configuration and performing more troubleshooting directly from the console.
In the Amazon ECS console, choose the Connect button to launch ECS Exec in CloudShell, as shown in the following figure. This capability needs the Amazon ECS service to have enableExecuteCommand set to true for the Connect button to be enabled.
Figure 3: ECS Exec Connect button in task overview page
When the CloudShell terminal appears, a pop-up window displays with the execute-command pre-filled and ready to use. Choosing Run initiates the exec session into the container, establishing a direct connection to the running task, as shown in the following figures. This immediate access eliminates the previous workflow that needed separate terminal sessions and multiple authentication steps.
Figure 4: ECS Exec run command in CloudShell
Figure 5: ECS Exec session in running container
With the session established in the container, we can now examine the application’s runtime configuration and confirm the database connection string being passed to the application. We check the environment variables to verify that the connection string in the environment matches exactly what was specified in the task definition, eliminating one potential cause of the issue, as shown in the following figure.
Figure 6: Database connection string in running container environment variables
To investigate further, we can examine the application’s source code to confirm how it reads environment variables and whether any defaults might be set at runtime.
Looking at the code confirms that the application correctly references the DATABASE_URL environment variable that is set in the task definition. This rules out the application code as the source of the problem, as shown in the following figure.
Figure 7: Database connection string in the application source code inside the running container
ECS Exec in CloudShell allows us to perform network diagnostics directly from within the container environment. We test the DNS resolution for the database endpoint and discover that the hostname cannot be resolved through DNS lookup within the container environment, as shown in the following figure. This reveals the root cause: the database endpoint provided to the application cannot be resolved, which explains why the connection attempts are failing.
Figure 8: DNS Lookup for database endpoint from inside the running container
Through this investigation, we’ve confirmed that the issue isn’t related to the application code, rather it’s related to a configuration problem—specifically an incorrect or invalid database endpoint specified in the task definition. The solution is clear: the application team should update the Amazon ECS task definition with the correct database endpoint and deploy an updated version of the Amazon ECS service. The entire troubleshooting workflow was completed efficiently within the Amazon ECS console. This demonstrates how these integrated tools streamline container management and accelerate problem resolution.
Cleaning up
If ECS Exec is no longer needed, then you can disable it by passing the --disable-execute-command flag in the UpdateService or RunTask API call.
The preceding command redeploys the tasks of the service.
Conclusion
The integration of Amazon CloudWatch Logs Live Tail and Amazon ECS Exec with AWS CloudShell into the Amazon ECS console directly addresses common customer pain points regarding visibility and access, particularly for containers running in production environments with restricted network access. The result is a more streamlined workflow that reduces the mean time to resolution (MTTR) and improves the overall developer experience with Amazon ECS.
To learn more about ECS Exec and CloudWatch Logs Live Tail, you can test these features using the Amazon ECS troubleshooting workshop How do I troubleshoot DNS resolution failures in Amazon ECS tasks? Scenario, or use these features to troubleshoot your application issues.
Get started with these new features today by updating your Amazon ECS configurations and exploring the new capabilities in the Amazon ECS console.
About the authors
Henrique Santana is a containers specialist that helps organizations modernize their technology stacks through container adoption and orchestration solutions. He’s guided numerous enterprises in overcoming containerization challenges, resulting in improvements in operational efficiency and accelerated time-to-market. When not optimizing container environments, Henrique shares insights from the frontlines of infrastructure to help businesses navigate their cloud-native journeys.
Nataizya Sikasote has a strong interest in containers (specifically ECS and Kubernetes) and hands-on development experience in Python and infrastructure as code using CloudFormation and CDK, he brings a comprehensive understanding of both the technical and operational aspects of modern container platforms. Nataizya enjoys enabling customers to build, deploy, and scale containerized applications effectively on AWS.
Stacey Hou is a Senior Product Manager – Technical at AWS, where she focuses on GenAI initiatives and observability for Amazon Elastic Container Service (ECS). She works closely with customers and engineering teams to drive innovations that simplify the experience of building, operating, and troubleshooting containerized applications.