Announcing additional Linux controls for Amazon ECS tasks on AWS Fargate
An Amazon Elastic Container Service (Amazon ECS) task is a number of co-located containers that are scheduled on to AWS Fargate or an Amazon EC2 container instance. Containers use Linux namespaces to provide workload isolation—and with namespaces—even though containers are scheduled together in an Amazon ECS task, they’re still isolated from each other and from the host.
Today we’re excited to announce that customers can now tune Linux kernel parameters in ECS tasks on AWS Fargate. Tuning Linux kernel parameters can help customers optimize their network throughput when running containerized network proxies or achieve higher levels of workload resilience by terminating stale connections. This launch provides parity for ECS tasks launched on AWS Fargate and Amazon EC2 container instances.
Additionally with today’s launch, the process id (PID) namespace can now be shared by all containers in the same ECS task on AWS Fargate. Sharing the PID namespace between containers in the same ECS task unlocks additional workload observability on AWS Fargate. Observability tools, such as container runtime security tools, can now run as a side car container and observe an application’s processes in the shared PID namespace. The PID namespace joins the network namespace, which is used with the awsvpc networking mode, in the list of namespaces that can be shared by all containers in an ECS task on AWS Fargate.
In this post, we’ll dive into System Controls and PID namespace sharing on AWS Fargate.
Within a Linux system, the parameters of the kernel can be tuned with the command line utility sysctl. When starting containers locally, for example with the Docker or finch command line interface, you can pass in the
--sysctl flag to change the kernel parameters. Within an ECS task, parameters can be defined with the
systemControl key in a Task Definition.
Customers running containerized network proxies on AWS Fargate have told us they often need to tune
net.* kernel parameters to allow their workloads to reach higher throughput demands. Frequently requested kernel parameters include the maximum number of queued connections with
net.core.somaxconn and the range of temporary ephemeral ports with
When architecting a workload for resiliency, customers have also told us that they would like to customize an ECS task’s TCP keep alive parameters on AWS Fargate. Configuring a short TCP keep alive timeout allows an application to detect network failures quickly, closing existing failed connections. Examples of when to tune the TCP keep alive include when a containerized workload is communicating to an Amazon Aurora PostgreSQL Cluster and when troubleshooting an Amazon VPC NAT Gateway.
Up until this launch, the
systemControl key was only available to customers running ECS tasks on EC2 container instances, but today this is now available for ECS tasks on AWS Fargate. An abstract of an example Amazon ECS Task Definition where two parameters are tuned is shown below:
Sharing the process id namespace
The process id namespace restricts what a process in a container can see. By default, a containerized process can only see processes in the same container, not the processes of the other co-located containers or the underlying host. A common use case to share a process id namespace is for observability tools. Container runtime security tools often run in a side car container and need to observe the processes in the application container. In this pattern, a process in the side car container monitors the processes in the application container to see if they start to make suspicious system calls.
With today’s launch, customers can now share a process id namespace among containers in the same ECS task by passing in the
pidMode key with the value
task in a Task Definition.
In this walkthrough we will start an ECS task with a shared process id namespace. The task contains two containers, an application container (nginx) and a sidecar container (a sleep process used as a demonstration). We will then show how a process in the sidecar container can interact with processes in the application container.
- An existing Amazon ECS cluster and Amazon VPC. If you need to create these in your AWS Account, see the Amazon ECS getting started guide.
- Ensure the ECS exec prerequisites are met in your AWS account and on the workstation you plan to execute the commands.
Run an ECS task on AWS Fargate
1. Create a Task Definition with two containers and
pidMode enabled. You will need to replace the IAM roles (
taskRoleArn) within the Task Definition with IAM roles created in the prerequisites.
2. Register the Task Definition with the
aws ecs register-task-definition command.
3. Run the Amazon ECS task on AWS Fargate with the
aws ecs run-task command. In the example below, replace the ECS cluster name, the VPC subnet and the security group values. As this task will not be externally accessed, a private VPC subnet and a security group with no ingress rules (such as the default VPC security group) will suffice.
4. Once the ECS task is running, use the
aws ecs execute-command command to create a terminal session in the sidecar container within the ECS task. If you receive an error when running this command, you can use the amazon-ecs-exec-checker script to ensure all of the prerequisites have been met.
5. Within the ECS exec terminal session we can now explore the shared PID namespace. To do so, we need to install some diagnostics tools inside of the sidecar container.
$ yum install procps strace -y
6. Using the ps command (included in the
procps package installed above) we can see all of the running processes in the shared PID namespace. The output shows the processes from the sidecar container as well as the nginx processes from the application container. The AWS Systems Managed Session Manager processes used to provide the ECS exec terminal are also shown.
7. Using the shared PID namespace we can monitor the system calls made by a process in the application container. We will use the
strace package installed in step five to monitor the main nginx process. To generate system calls, we will forcefully stop a nginx worker process with the
kill command. In my case the main nginx process is process ID 7 and a worker process is process ID 56, these will be different in your environment and need to be replaced in the commands below.
In this walkthrough we have shown that by sharing a process id namespace, a process in a sidecar container can now interact and observe with all running processes in all containers in the ECS task. In step seven we used a sidecar container to monitor and forcefully stop processes in the application container.
To clean up this walkthrough:
1. Exit the ECS exec terminal by running the
exit command in the terminal window with the open session.
2. Stop the ECS task with the
aws ecs stop-task command.
3. Deregister the ECS task definition with the
aws ecs deregister-task-definition command.
Caveats of sharing a process id namespace
While a process id namespace can now be shared by containers in the same ECS task on AWS Fargate, there a few things to be aware of. We’ll walk through those caveats in the context of the application and sidecar ECS task defined previously:
- A process in the sidecar container can observe, stop, or restart a process in the application container.
- A process in the sidecar container can view the filesystem of the application container. For example, if the application is running as process ID 7, then within the sidecar container you can access the application containers filesystem at
/proc/7/root/. The only protection of the application containers filesystem would be done through Unix file permissions.
- When sharing a process in an ECS task, a new
pauseprocess runs as PID 1 for the whole task.
SYS_PTRACELinux capability may need to be added to the ECS task to provide full traceability of a processes running in the application container.
With this launch we are excited to unblock more workloads on AWS Fargate. Sharing a process id namespace and tuning kernel parameters are both features that were requested through the AWS Containers Roadmap for AWS Fargate. We value your feedback, and welcome you to submit any additional feature requests or improvements as GitHub issues on the roadmap. For more information on the AWS Fargate security architecture see the AWS Fargate Security Whitepaper.