Extending Amazon ECS Express Mode to Build an Optimal Container Environment

When you deploy containers with Amazon Elastic Container Service (Amazon ECS) Express Mode, Amazon ECS provides load balancing, scaling, logging, and networking without additional configuration. As your workload evolves, you may want to tailor this environment to your specific requirements. For example, you can access running containers with ECS Exec to troubleshoot issues, provision capacity ahead of predictable traffic spikes, or route application logs to the destination of your choice with FireLens. Because Express Mode creates standard AWS resources in your account, you can extend Express Mode using the same ECS APIs and other AWS service APIs you already use. This applies whether you are migrating from AWS App Runner or starting fresh.

Express Mode also supports custom task definitions, allowing you to apply advanced task-level configurations such as sidecar containers, custom health checks, and FireLens for custom log routing, while keeping the simplified deployment experience.

In this post, you walk through how to extend an Amazon ECS Express Mode service beyond its defaults, with three hands-on examples: turning on ECS Exec, adding a scheduled scaling policy, and adding a FireLens log router as a sidecar container using a custom task definition. You also learn which service-level settings Express Mode restricts and how to decide whether those restrictions affect your workload.

Solution overview

This section provides an overview of ECS Express Mode, the resources that Express Mode creates automatically, and the settings that you cannot change within it.

Overview of ECS Express Mode

Express Mode deploys a containerized application onto an ECS service with pre-configured defaults by specifying three required parameters which are container image URL, task execution role and infrastructure role.

When you use Express Mode, it creates these resources with default configurations to help you get started quickly. You can use these defaults to deploy a containerized application:

• An ECS default cluster with the AWS Fargate capacity provider (if one does not already exist)
• A task definition with container, logging, and network settings.
• A service with canary deployment and auto scaling.
• An Application Load Balancer (ALB) with an HTTPS listener, listener rule, and target group.
• Security groups with required ingress rules.
• Service-linked roles for auto scaling and load balancing.
• An AWS Application Auto Scaling scalable target and a target tracking scaling policy.
• An Amazon CloudWatch log group dedicated to the service.
• Metric alarms to detect failed deployments.
• An AWS Certificate Manager (ACM) certificate.

The following figure shows the relationships between resources created in ECS Express Mode.

Architecture of resources created by an ECS Express Mode service

An Internet-facing ALB, ECS clusters, ECS services, and security groups for the ALB and ECS tasks are created within the virtual private cloud (VPC), and tasks are launched within the ECS service. The tasks’ standard output logs are stored in Amazon CloudWatch Logs. Additionally, based on CloudWatch metric alarms, Application Auto Scaling adjusts the number of tasks as the CPUUtilization value increases.

Extending ECS Express Mode

When you use Express Mode, it creates standard AWS resources in your account. You can modify these resources using the ECS API and other AWS service APIs. For example:

• Add a task role to turn on ECS Exec or grant application-level permissions.
• Configure additional scaling policies such as scheduled or predictive scaling.
• Customize ALB listener rules, attach AWS WAF, or add a custom domain.
• Adjust Amazon CloudWatch Logs retention or add subscription filters.

In addition to modifying these resources, Express Mode supports custom task definitions. You can provide your own task definition to apply advanced task-level configurations, such as sidecar containers, custom container health checks, and FireLens for custom log routing. The walkthrough section uses this capability to add a FireLens log router as a sidecar container.

For the full list of resources and their defaults, refer to the Resources created by Amazon ECS Express Mode services documentation.

Express Mode restricts two ECS service-level settings:

1. deploymentConfiguration — Express Mode uses canary deployment exclusively. You cannot switch to rolling update or blue/green deployment. If your release process requires rolling update or blue/green deployment, create a standard ECS service instead.
2. loadBalancers — Express Mode provisions an ALB for each service. You cannot replace it with a Network Load Balancer (NLB) or remove it entirely. If your workload requires TCP/UDP passthrough, gRPC without HTTP/2 termination, or operates without a load balancer, create a standard ECS service instead.

You can customize all other service and task-level settings through the standard ECS API. The walkthrough section demonstrates three common extensions.

Prerequisites

Before you begin, verify that you have the following prerequisites:

• An AWS account with appropriate permissions.
• AWS Command Line Interface (AWS CLI) (version 2.0 or later) and jq installed and configured.
• Basic familiarity with Amazon ECS service concepts.

Walkthrough

In this section, you create an ECS Express Mode service and extend it with three configurations:

• Turning on ECS Exec.
• Adding a scheduled scaling policy.
• Adding a FireLens container.

Creating an ECS Express Mode service

1. Set the environment variables used throughout these steps.

export AWS_REGION="us-east-2"
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

2. Create a task execution role and attach the managed policy.

aws iam create-role --role-name ecsTaskExecutionRole \
    --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"Service": "ecs-tasks.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }]
    }'

aws iam attach-role-policy --role-name ecsTaskExecutionRole \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

3. Create an infrastructure role and attach the managed policy.

aws iam create-role --role-name ecsInfrastructureRoleForExpressServices \
    --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "AllowAccessInfrastructureForECSExpressServices",
            "Effect": "Allow",
            "Principal": {"Service": "ecs.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }]
    }'

aws iam attach-role-policy --role-name ecsInfrastructureRoleForExpressServices \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSInfrastructureRoleforExpressGatewayServices

4. Create an Express Mode service. This example uses the nginx image. Express Mode uses the default cluster and creates one if it does not exist.

SERVICE_NAME=$(aws ecs create-express-gateway-service \
    --region ${AWS_REGION} \
    --primary-container "image"="public.ecr.aws/nginx/nginx:latest" \
    --execution-role-arn arn:aws:iam::${ACCOUNT_ID}:role/ecsTaskExecutionRole \
    --infrastructure-role-arn arn:aws:iam::${ACCOUNT_ID}:role/ecsInfrastructureRoleForExpressServices \
    --query "service.serviceName" --output text)

export CLUSTER_NAME="default"
aws ecs wait services-stable \
    --region ${AWS_REGION} \
    --cluster default \
    --services ${SERVICE_NAME}

Turning on ECS Exec

Express Mode does not assign a task role by default.

ECS Exec requires a task role with ssmmessages permissions because the AWS Systems Manager Agent (SSM Agent) inside the container connects to AWS Systems Manager Session Manager.

This example also configures the cluster to store Exec session output in CloudWatch Logs.

1. Create a log group for ECS Exec session output.

aws logs create-log-group \
    --region ${AWS_REGION} \
    --log-group-name /ecs/exec-output/${CLUSTER_NAME}

2. Add ECS Exec logging configuration to the cluster.

aws ecs update-cluster \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --configuration '{
        "executeCommandConfiguration": {
            "logging": "OVERRIDE",
            "logConfiguration": {
                "cloudWatchLogGroupName": "/ecs/exec-output/'"${CLUSTER_NAME}"'",
                "cloudWatchEncryptionEnabled": false
            }
        }
    }'

3. Create a task role and attach the required permissions.

aws iam create-role --role-name ecsExecTaskRole \
    --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"Service": "ecs-tasks.amazonaws.com"},
            "Action": "sts:AssumeRole"
        }]
    }'

aws iam put-role-policy --role-name ecsExecTaskRole \
    --policy-name ECSExecPolicy \
    --policy-document '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ssmmessages:CreateControlChannel",
                    "ssmmessages:CreateDataChannel",
                    "ssmmessages:OpenControlChannel",
                    "ssmmessages:OpenDataChannel"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "logs:CreateLogStream",
                    "logs:PutLogEvents"
                ],
                "Resource": "arn:aws:logs:*:*:log-group:/ecs/exec-output/*"
            }
        ]
    }'

4. Add the task role to the Express Mode service.

This updates the task definition associated with the Express Mode service to include the task role:

SERVICE_ARN=$(aws ecs describe-services \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --services ${SERVICE_NAME} \
    --query "services[0].serviceArn" --output text)

aws ecs update-express-gateway-service \
    --region ${AWS_REGION} \
    --service-arn ${SERVICE_ARN} \
    --task-role-arn arn:aws:iam::${ACCOUNT_ID}:role/ecsExecTaskRole

Wait for the deployment to complete. The service must reach a steady state before proceeding to the next step.

aws ecs wait services-stable \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --services ${SERVICE_NAME}

At this point, the task definition includes the required task role, but ECS Exec itself is still turned off at the service level. The next step turns on ECS Exec and forces a new deployment so that newly launched tasks start the SSM Agent with the permissions you configured.

5. Turn on ECS Exec and trigger a new deployment.

Use update-service to turn on ECS Exec. Only tasks launched after you turn on ECS Exec receive the Exec capability. Pass the --force-new-deployment flag to replace existing tasks with new ones that include the SSM Agent.

aws ecs update-service \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --service ${SERVICE_NAME} \
    --enable-execute-command \
    --force-new-deployment

Wait for the service to reach a steady state:

aws ecs wait services-stable \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --services ${SERVICE_NAME}

6. Verify that ECS Exec is active.

Retrieve a task ID from the service and confirm that the ExecuteCommandAgent lastStatus is RUNNING.

TASK_ARN=$(aws ecs list-tasks \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --service-name ${SERVICE_NAME} \
    --query "taskArns[0]" --output text)

aws ecs describe-tasks \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --tasks ${TASK_ARN} \
    --query "tasks[0].containers[?name=='Main'].managedAgents" --output table

7. Connect to a container inside the task.

aws ecs execute-command \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --task ${TASK_ARN} \
    --container "Main" \
    --interactive \
    --command "/bin/sh"

You have now turned on ECS Exec for the Express Mode service. ECS stores session logs in the /ecs/exec-output/<cluster-name> log group.

Adding a scheduled scaling policy

Express Mode creates a target tracking scaling policy by default. This policy scales reactively based on CPUUtilization metric changes. When your traffic pattern is predictable, scheduled scaling helps you provision capacity ahead of demand.

In this example, you configure two schedules. The first raises the minimum task count to 5 at 8:00 AM (Asia/Tokyo) on weekdays. The second resets the minimum to 1 at 10:00 PM.

1. Create the scale-out schedule.

aws application-autoscaling put-scheduled-action \
    --region ${AWS_REGION} \
    --service-namespace ecs \
    --resource-id service/${CLUSTER_NAME}/${SERVICE_NAME} \
    --scalable-dimension ecs:service:DesiredCount \
    --scheduled-action-name scale-out-weekday-morning \
    --schedule "cron(0 8 ? * MON-FRI *)" \
    --timezone "Asia/Tokyo" \
    --scalable-target-action MinCapacity=5,MaxCapacity=10

2. Create the scale-in schedule.

aws application-autoscaling put-scheduled-action \
    --region ${AWS_REGION} \
    --service-namespace ecs \
    --resource-id service/${CLUSTER_NAME}/${SERVICE_NAME} \
    --scalable-dimension ecs:service:DesiredCount \
    --scheduled-action-name scale-in-weekday-night \
    --schedule "cron(0 22 ? * MON-FRI *)" \
    --timezone "Asia/Tokyo" \
    --scalable-target-action MinCapacity=1,MaxCapacity=10

3. Confirm the schedules are registered.

aws application-autoscaling describe-scheduled-actions \
    --region ${AWS_REGION} \
    --service-namespace ecs \
    --resource-id service/${CLUSTER_NAME}/${SERVICE_NAME}

Verify that the output contains scale-out-weekday-morning and scale-in-weekday-night.

Adding a FireLens container

By default, Express Mode sends application logs to Amazon CloudWatch Logs using the awslogs log driver. FireLens gives you more flexible log management, such as filtering logs and routing them to multiple destinations. In this section, you use a custom task definition to add a Fluent Bit FireLens container as a sidecar and route application logs to a dedicated log group.

With a custom task definition, Express Mode uses your task definition as-is and continues to manage the infrastructure, including load balancing, auto scaling, and canary deployments. When you use a custom task definition, you cannot pass the task role through the Express Mode API. Instead, Express Mode derives it from the taskRoleArn field in your task definition.

1. Create the log groups for FireLens.

Create two log groups: one for the application logs that FireLens routes, and one for the logs of the Fluent Bit container itself.

aws logs create-log-group \
    --region ${AWS_REGION} \
    --log-group-name /ecs/firelens/application

aws logs create-log-group \
    --region ${AWS_REGION} \
    --log-group-name /ecs/firelens/log-router

2. Add permissions for the FireLens log destination to the task role.

FireLens (Fluent Bit) needs permissions to send logs to CloudWatch Logs. Add an inline policy to the task role that you created in the previous section.

aws iam put-role-policy --role-name ecsExecTaskRole \
    --policy-name FireLensLogPolicy \
    --policy-document '{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:log-group:/ecs/firelens/*"
        }]
    }'

3. Retrieve the current task definition.

Create a new revision based on the task definition that Express Mode currently manages. The task definition ARN is available in the PRIMARY deployment of the service.

TASK_DEF_ARN=$(aws ecs describe-services \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --services ${SERVICE_NAME} \
    --query "services[0].deployments[?status=='PRIMARY'] | [0].taskDefinition" \
    --output text)

aws ecs describe-task-definition \
    --region ${AWS_REGION} \
    --task-definition ${TASK_DEF_ARN} \
    --query "taskDefinition" > task-definition.json

4. Add the FireLens container to the task definition.

The following command makes three changes to the task definition and saves the result as new-task-definition.json:

• Adds a log_router container with the firelensConfiguration field, which configures it as a FireLens container. The awslogs log driver stores the logs of the Fluent Bit container itself in a separate log group.
• Changes the logConfiguration of the Main container to awsfirelens so that FireLens routes the application logs to the /ecs/firelens/application log group and reduces its memoryReservation from 2048 to 1998. This keeps the total memory reservation of the two containers within the task memory (2048 MiB).
• Removes the read-only fields that register-task-definition does not accept.

  jq --arg region "${AWS_REGION}" '
  .containerDefinitions += [{
    "name": "log_router",
    "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
    "essential": true,
    "memoryReservation": 50,
    "firelensConfiguration": { "type": "fluentbit" },
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/firelens/log-router",
        "awslogs-region": $region,
        "awslogs-stream-prefix": "firelens"
      }
    }
  }]
  | (.containerDefinitions[] | select(.name == "Main")).logConfiguration = {
      "logDriver": "awsfirelens",
      "options": {
        "Name": "cloudwatch_logs",
        "region": $region,
        "log_group_name": "/ecs/firelens/application",
        "log_stream_prefix": "app-"
      }
    }
  | (.containerDefinitions[] | select(.name == "Main")).memoryReservation = 1998
  | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy)
' task-definition.json > new-task-definition.json

5. Register the new task definition.

 NEW_TASK_DEF_ARN=$(aws ecs register-task-definition \
    --region ${AWS_REGION} \
    --cli-input-json file://new-task-definition.json \
    --query "taskDefinition.taskDefinitionArn" --output text)

6. Update the Express Mode service with the custom task definition.

Pass the registered task definition to the --task-definition-arn parameter of update-express-gateway-service.

aws ecs update-express-gateway-service \
    --region ${AWS_REGION} \
    --service-arn ${SERVICE_ARN} \
    --task-definition-arn ${NEW_TASK_DEF_ARN}

Wait until the service reaches a steady state.

aws ecs wait services-stable \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --services ${SERVICE_NAME}

7. Verify the log routing.

After the deployment completes, confirm that FireLens delivers the application logs to the log group.

aws logs tail /ecs/firelens/application --region ${AWS_REGION} --since 5m

You can also confirm the logs of the Fluent Bit container itself in the /ecs/firelens/log-router log group.

aws logs tail /ecs/firelens/log-router --region ${AWS_REGION} --since 5m

Your application logs now flow to the /ecs/firelens/application log group instead of the log group that Express Mode created by default. Express Mode does not overwrite this change unless you request a conflicting update through the Express Mode APIs, such as specifying a new log group name.

Clean up

To avoid ongoing charges, delete the resources you created in this walkthrough.

1. Delete the scheduled scaling actions.

aws application-autoscaling delete-scheduled-action \
    --region ${AWS_REGION} \
    --service-namespace ecs \
    --resource-id service/${CLUSTER_NAME}/${SERVICE_NAME} \
    --scalable-dimension ecs:service:DesiredCount \
    --scheduled-action-name scale-out-weekday-morning

aws application-autoscaling delete-scheduled-action \
    --region ${AWS_REGION} \
    --service-namespace ecs \
    --resource-id service/${CLUSTER_NAME}/${SERVICE_NAME} \
    --scalable-dimension ecs:service:DesiredCount \
    --scheduled-action-name scale-in-weekday-night

2. Delete the Express Mode service.

Running delete-express-gateway-service removes all Express Mode-managed resources: the ECS service, ALB, target group, listener rules, security groups, ACM certificate, auto scaling configuration, and CloudWatch alarms.

aws ecs delete-express-gateway-service \
    --region ${AWS_REGION} \
    --service-arn ${SERVICE_ARN}

3. Deregister the task definitions for all revisions.

Deregister all revisions of the task definition family, including the revisions that Express Mode created. Run this after you delete the Express Mode service.

TASK_DEF_FAMILY=$(echo ${NEW_TASK_DEF_ARN} | sed 's|.*task-definition/||; s|:.*||')

for TD_ARN in $(aws ecs list-task-definitions \
    --region ${AWS_REGION} \
    --family-prefix ${TASK_DEF_FAMILY} \
    --query "taskDefinitionArns[]" --output text); do
    aws ecs deregister-task-definition \
        --region ${AWS_REGION} \
        --task-definition ${TD_ARN} > /dev/null
done

4. Delete the log groups for ECS Exec session output and FireLens.

aws logs delete-log-group \
    --region ${AWS_REGION} \
    --log-group-name /ecs/exec-output/${CLUSTER_NAME}
aws logs delete-log-group \
    --region ${AWS_REGION} \
    --log-group-name /ecs/firelens/application
aws logs delete-log-group \
    --region ${AWS_REGION} \
    --log-group-name /ecs/firelens/log-router

5. Remove the Exec logging configuration from the cluster.

aws ecs update-cluster \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME} \
    --configuration '{
        "executeCommandConfiguration": {
            "logging": "DEFAULT"
        }
    }'

6. Delete the IAM roles.

Delete the task role inline policy and the role itself:

aws iam delete-role-policy --role-name ecsExecTaskRole --policy-name ECSExecPolicy
aws iam delete-role-policy --role-name ecsExecTaskRole --policy-name FireLensLogPolicy
aws iam delete-role --role-name ecsExecTaskRole

Detach the managed policy from the task execution role and delete the role:

aws iam detach-role-policy --role-name ecsTaskExecutionRole \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name ecsTaskExecutionRole

Detach the managed policy from the infrastructure role and delete the role:

aws iam detach-role-policy --role-name ecsInfrastructureRoleForExpressServices \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSInfrastructureRoleforExpressGatewayServices
aws iam delete-role --role-name ecsInfrastructureRoleForExpressServices

7. Delete the ECS cluster (optional).

If you no longer need the default cluster that Express Mode created, delete it:

aws ecs delete-cluster \
    --region ${AWS_REGION} \
    --cluster ${CLUSTER_NAME}

Conclusion

In this post, you reviewed the resources that Express Mode creates automatically and identified the settings that you cannot change within Express Mode. You then extended an Express Mode service through three hands-on examples: turning on ECS Exec, adding a scheduled scaling policy, and adding a FireLens container as a sidecar. You can extend your ECS Express Mode service based on your requirements.

To explore the full capabilities of Amazon ECS Express Mode, refer to the Amazon ECS Express Mode documentation. For the full list of Express Mode default configurations, refer to the Resources created by Amazon ECS Express Mode services documentation. To learn more about using custom task definitions with Express Mode, refer to the Create an Express Mode service with your own task definition and Updating with a custom task definition documentation. For additional extension examples, refer to Updating resources outside of Express Mode documentation.

About the author