Maintaining Transport Layer Security all the way to your container: using the Application Load Balancer with Amazon ECS and Envoy

This post is contributed by Sri Mallu, Re Alvarez-Parmar, and Sahil Thapar

Application Load Balancer has been an instrumental element when it comes to building highly available, scalable, and secure applications. AWS customers can rely on ALB to perform functions that have been traditionally implemented in application’s code. Let’s take connection security as an example, ALB can be used to offload the work of encryption and decryption so that your applications can focus on business logic.

The Application Load Balancer allows you to create an HTTPS listener, which uses encrypted connections (also known as SSL offload). This feature enables traffic encryption between your load balancer and the clients that initiate SSL or TLS sessions. When you create an HTTPS listener, you deploy a SSL/TLS server certificate on your load balancer. The load balancer uses this server certificate to terminate the front-end connection and then decrypt requests from clients before sending them to the targets. Why does ALB need to decrypt requests?

The reason why ALB needs to decrypt the request is because it operates at the application layer of the Open Systems Interconnection (OSI) model and needs to inspect the requests to perform request routing. You can use ALB to route requests based on HTTP headers, methods, query parameters, and source IP CIDRs. That’s why when you use ALB to load balance your applications, SSL/TLS termination is done at ALB, and typically the connection between ALB and the backend application is left unencrypted. Terminating secure connections at the load balancer and using HTTP on the backend might be sufficient for your application. Network traffic between AWS resources can only be listened to by the instances that are part of the connection.

However, if you are developing an application that needs to comply with strict external regulations, you might be required to secure all network connections.

I am going to show you how to encrypt connection between clients and the load balancer and from the load balancer to your application container running in an ECS cluster. In this post, we run ECS containers on Fargate but the solution applies to ECS containers running on EC2 as well. To follow along you will need a VPC with public and private subnets, appropriate route tables, an internet gateway, and NAT gateway(s).

We will also use Envoy as a front proxy that terminates TLS and we will run Envoy as a sidecar along with the application container. With this method, we do not need to handle encryption in the application code. Envoy will send traffic, unencrypted, to the application container over localhost.

Prerequisites

In order to successfully carry out steps outlined:

• Make sure to have latest AWS CLI installed and configured.
• Have Docker and OpenSSL installed.

Architecture

?You may use this template as a base line for the setup.

We will start by creating a self-signed certificate, this will be embedded into the Envoy container image. Then we will create an ACM certificate using the same private key and certificate and import it into ACM. We will use this ACM certificate on the ALB as the server certificate.

Even though this solution uses a self-signed certificate, you can use an ACM private CA to generate the certificate, as well. You cannot use ACM public Certificate Authority (CA) for this solution, as it does not allow you to export private keys. However, you can use any commercially available trusted CA that exports private keys.

?Since we are using a self-signed certificate, if you access the application from a browser, you will need to set up the trust in the browser. This is not required if you use a commercial trusted CA.

Tutorial

Let’s define a few environment variables that will be used throughout the tutorial.

##Export region and account
export account=<account> # <- Your account number here
export AWS_REGION=<AWS Region> # <- Your AWS Region

##Export key networking constructs
export private_subnet1="subnet-0cb23e7b2da6116ec" # Subsitute these values with your VPC subnet ids
export private_subnet2="subnet-01446062d07790b98"
export public_subnet1="subnet-092e36d83b9d0fd51"
export public_subnet2="subnet-0a41fcbbe7add4b76"
export sg=sg-0ea3f8730146cc784   ##Wide open SG for ALB All All 0.0.0.0/0
export vpcId=vpc-0a0598d1d7d1dd8b3 # <- Change this to your VPC id

##Service name and domain to be used
export service_name=ecs-encryption
export dns_namespace=awsblogs.info  ##This should be your domain

If you are using the CloudFormation template mentioned above, an ECS task execution IAM role will be created for you. If you are creating the role yourself, please verify that it has the following permissions and trust policy.

##Policy
{
    "Statement": [
        {
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

##Trust
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create a ecsTaskExecutionRoleArn environment variable.

export ecsTaskExecutionRoleArn=arn:aws:iam::551961765653:role/ECS-ENCRYPTION-ECSTaskExecutionRole-URQRCO2HC4E3

ECR repository setup

Create two ECR repositories to store the application and Envoy container images.

Repository 1:

aws ecr create-repository \
--repository-name ${service_name}-blog-app \
--region $region

Output:

{
    "repository": {
        "repositoryArn": "arn:aws:ecr:us-west-1:<account>:repository/ecs-end-end-encryption-blog-app",
        "registryId": "<account>",
        "repositoryName": "ecs-end-end-encryption-blog-app",
        ....
}

Repository 2:

export aws_ecr_repository_url_app=$account.dkr.ecr.$region.amazonaws.com/${service_name}-blog-app

aws ecr create-repository --repository-name ${service_name}-blog-proxy \
--region $region

Output:

{
    "repository": {
        "repositoryArn": "arn:aws:ecr:us-west-1:<account>:repository/ecs-end-end-encryption-blog-proxy",
        "registryId": "<account>",
        "repositoryName": "ecs-end-end-encryption-blog-proxy",
        ...
}

export aws_ecr_repository_url_proxy=$account.dkr.ecr.$region.amazonaws.com/${service_name}-blog-proxy

Certificate setup

Let’s create a key pair and import it as an ACM certificate. We will associate this certificate with the ALB.
The same key and certificate will be used to enable TLS encryption in the Envoy proxy.

mkdir -p docker/certs && cd docker/certs

##Create the config
cat <<EOF > castore.cfg
[ req ]
default_bits = 2048
default_keyfile = my-aws-private.key
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[ req_distinguished_name ]
C = US
ST = VA
L = Richmond
O = awsblogs.info
OU = awsblogs.info
CN= ecs-encryption.awsblogs.info ## Use your domain
emailAddress = user@email.com ## Use your email address
[v3_ca]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = CA:true
[v3_req]
## Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
EOF

Use OpenSSL to create the certificate signing authority and then generate the private key and certificate using it.

openssl genrsa -out castore.key 2048
openssl req -x509 -new -nodes -key castore.key -days 3650 -config castore.cfg -out castore.pem

openssl genrsa -out my-aws-private.key 2048
openssl req -new -key my-aws-private.key -out my-aws.csr -config castore.cfg
openssl x509 -req -in my-aws.csr -CA castore.pem -CAkey castore.key -CAcreateserial  -out my-aws-public.crt -days 365

aws acm import-certificate \
--certificate file://my-aws-public.crt \
--private-key file://my-aws-private.key \
--certificate-chain  file://castore.pem \
--region $region

Output:

{
    "CertificateArn": "arn:aws:acm:us-west-1:<account>:certificate/838bca7e-d33b-41e1-b590-97253cd6e506"
}

Export the certificate Arn.

export certificateArn=arn:aws:acm:us-west-1:<account>:certificate/838bca7e-d33b-41e1-b590-97253cd6e506

Let’s verify in the ACM console that the certificate was imported.

Build Docker images and push them to ECR

Let’s create two Docker images, one for Envoy proxy and another for a demo hello application. We will later create a task definition will have this pair of containers defined. Envoy proxy will run as sidecar and route requests to the hello application container over localhost.

Define a simple hello service, with following content:

cd docker

cat <<EOF > service.py
from flask import Flask
import socket

app = Flask(__name__)

@app.route('/service')
def hello():
  return (f'Hello from behind Envoy proxy!!\n')

if __name__ == "__main__":
  app.run(host='0.0.0.0', port=8080, debug=True)
EOF

Create a Dockerfile for the application container.

cat <<EOF > Dockerfile-app
FROM envoyproxy/envoy-alpine-dev:latest

RUN apk update && apk add python3 bash curl; \
      pip3 install -q Flask==0.11.1 requests==2.18.4; \
      mkdir /code

ADD ./service.py /code

EXPOSE 8080

CMD ["python3", "/code/service.py"]
EOF

Create an Envoy configuration file. It will used for proxying and routing requests. All requests from ALB will encrypted using TLS. The proxy will route requests to the application container over HTTP.

cat <<EOF > envoy.yaml
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    filter_chains:
      tls_context:
        common_tls_context:
          tls_certificates:
          - certificate_chain:
              filename: "/etc/ssl/my-aws-public.crt"
            private_key:
              filename: "/etc/ssl/my-aws-private.key"
      filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: service
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: local_service
          http_filters:
          - name: envoy.router
            config: {}
  clusters:
  - name: local_service
    connect_timeout: 0.5s
    type: strict_dns
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: 127.0.0.1
        port_value: 8080

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8081
EOF

Create a startup script to run Envoy.

cat <<EOF > start_envoy.sh
#!/bin/sh
/usr/local/bin/envoy -c /etc/envoy.yaml
EOF

Create a Dockerfile for the Envoy proxy.

cat <<EOF > Dockerfile-proxy
FROM envoyproxy/envoy-dev:latest

RUN apt-get update && apt-get -q install -y \
    curl wget jq python \
        python-pip \
        python-setuptools \
        groff \
        less \
        && pip --no-cache-dir install --upgrade awscli
RUN mkdir -p /etc/ssl
ADD start_envoy.sh /start_envoy.sh
ADD envoy.yaml /etc/envoy.yaml
ADD certs /etc/ssl/

RUN chmod +x /start_envoy.sh

ENTRYPOINT ["/bin/sh"]
EXPOSE 443
CMD ["start_envoy.sh"]
EOF

Let’s build the Docker images and push them to ECR.

## Build images locally, make sure you are in the docker folder
docker build -t ${aws_ecr_repository_url_proxy} -f Dockerfile-proxy .
docker build -t ${aws_ecr_repository_url_app} -f Dockerfile-app .

## Verify
docker images | grep $region
 
## Login to ECR
aws ecr get-login --region $region

## Grab the password from output of previous command and execute the following
docker login -u AWS -p <password> https://$account.dkr.ecr.$region.amazonaws.com
Login Succeeded

## Push to ECR
docker push ${aws_ecr_repository_url_proxy}
docker push ${aws_ecr_repository_url_app}

Verify that the images have been pushed to ECR.

Create cluster and task definition

Create a task definition with both the container definitions.

cat <<EOF >  ecs_task_def.template
{
   "containerDefinitions": [
      {
       "logConfiguration": {
         "logDriver": "awslogs",
         "options": {
           "awslogs-group": "/ecs/$service_name",
           "awslogs-region": "$region",
           "awslogs-stream-prefix": "ecs"
         }
       },
       "portMappings": [
         {
           "hostPort": 443,
           "protocol": "tcp",
           "containerPort": 443
         }
       ],
       "cpu": 0,
       "environment": [
         {"name":  "DNS_NAME", "value":  "$service_name.awsblogs.info"}
       ],
       "image": "$aws_ecr_repository_url_proxy:latest",
       "name": "envoy"
     },
     {
       "logConfiguration": {
         "logDriver": "awslogs",
         "options": {
           "awslogs-group": "/ecs/$service_name",
           "awslogs-region": "$region",
           "awslogs-stream-prefix": "ecs"
         }
       },
       "cpu": 0,
       "image": "$aws_ecr_repository_url_app:latest",
       "name": "service"
     }
   ],
   "cpu": "256",
   "taskRoleArn": "$ecsTaskExecutionRoleArn",
   "executionRoleArn": "$ecsTaskExecutionRoleArn",
   "family": "$service_name",
   "memory": "512",
   "networkMode": "awsvpc",
   "requiresCompatibilities": [ 
       "FARGATE" 
    ]

}
EOF

Substitute the environment variables, create a log group, an ECS cluster, and register the task definition.

envsubst <ecs_task_def.template>ecs_task_def.json

export log_group_name=/ecs/ecs-end-end-encryption

aws logs create-log-group \
--log-group-name $log_group_name \
--region $region

export cluster=${service_name}-cluster

aws ecs create-cluster --cluster-name $cluster \
--region $region

Output:

{
    "cluster": {
        "clusterArn": "arn:aws:ecs:us-west-1:<account>:cluster/ecs-encryption-cluster",
        "clusterName": "ecs-encryption-cluster",
        "status": "ACTIVE",
        "registeredContainerInstancesCount": 0,
        "runningTasksCount": 0,
        "pendingTasksCount": 0,
        "activeServicesCount": 0,
        "statistics": [],
        "tags": [],
        "settings": [
            {
                "name": "containerInsights",
                "value": "disabled"
            }
        ],
        "capacityProviders": [],
        "defaultCapacityProviderStrategy": []
    }
}

Register the task definition.

aws ecs register-task-definition \
--cli-input-json file://ecs_task_def.json \
--region=$region

Verify the creation of the task definition on the AWS Management Console.

Create the Application Load Balancer

Create the Application Load Balancer and setup listener rules and target groups for the application.

export alb=${service_name}-alb

aws elbv2 create-load-balancer --name $alb \
--scheme internet-facing \
--subnets $public_subnet1 $public_subnet2 \
--security-groups $sg --region $region

Output:

{
    "LoadBalancers": [
        {
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:us-west-1:<account>:loadbalancer/app/ecs-encryption-alb/420cd49c9b77c43b",
            "DNSName": "ecs-encryption-alb-860337044.us-west-1.elb.amazonaws.com",
            ....
        }
    ]
}

export loadbalancerArn=arn:aws:elasticloadbalancing:us-west-1:<account>:loadbalancer/app/ecs-encryption-alb/420cd49c9b77c43b 

aws elbv2 create-target-group \
--name https-target \
--protocol HTTPS \
--port 443 \
--health-check-path /service \
--target-type ip \
--vpc-id $vpcId \
--region $region

Output:

{
    "TargetGroups": [
        {
            "TargetGroupArn": "arn:aws:elasticloadbalancing:us-west-1:<account>:targetgroup/https-target/3e6bed06eed53f98",
            "TargetGroupName": "https-target",
            "Protocol": "HTTPS",
            "Port": 443,
            "VpcId": "vpc-0a0598d1d7d1dd8b3",
            ...
    ]
}

export targetGroupArn=arn:aws:elasticloadbalancing:us-west-1:<account>:targetgroup/https-target/3e6bed06eed53f98

aws elbv2 create-listener --load-balancer-arn $loadbalancerArn \
--protocol HTTPS --port 443  \
--certificates CertificateArn=$certificateArn \
--default-actions Type=forward,TargetGroupArn=$targetGroupArn \
--region $region

Output:

{
    "Listeners": [
        {
            "ListenerArn": "arn:aws:elasticloadbalancing:us-west-1:<account>:listener/app/ecs-encryption-alb/420cd49c9b77c43b/c234e7efab3456d9",
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:us-west-1:<account>:loadbalancer/app/ecs-encryption-alb/420cd49c9b77c43b",
            "Port": 443,
            "Protocol": "HTTPS",
            "Certificates": [
                {
                    "CertificateArn": "arn:aws:acm:us-west-1:<account>:certificate/838bca7e-d33b-41e1-b590-97253cd6e506"
                }
            ],
            "SslPolicy": "ELBSecurityPolicy-2016-08",
            "DefaultActions": [
                {
                    "Type": "forward",
                    "TargetGroupArn": "arn:aws:elasticloadbalancing:us-west-1:<account>:targetgroup/https-target/3e6bed06eed53f98",
                    "ForwardConfig": {
                        "TargetGroups": [
                            {
                                "TargetGroupArn": "arn:aws:elasticloadbalancing:us-west-1:<account>:targetgroup/https-target/3e6bed06eed53f98",
                                "Weight": 1
                            }
                        ],
                        "TargetGroupStickinessConfig": {
                            "Enabled": false
                        }
                    }
                }
            ]
        }
    ]
}

Create the service

Create the ECS service definition template. Replace the values in the file to match your account.

cat <<EOF >  ecs_service_def.template
{
    "serviceName": "$service_name-service",
    "cluster": "arn:aws:ecs:$region:$account:cluster/$cluster",
    "taskDefinition": "arn:aws:ecs:$region:$account:task-definition/$service_name",
    "loadBalancers": [
                {
                    "targetGroupArn": "$targetGroupArn",
                    "containerName": "envoy",
                    "containerPort": 443
                }
            ],
    "launchType": "FARGATE",
    "platformVersion": "LATEST",
    "networkConfiguration": {
                "awsvpcConfiguration": {
                    "subnets": [
                        "$private_subnet1", "$private_subnet2"
                    ],
                    "securityGroups": [
                        "$sg"
                    ],
                    "assignPublicIp": "ENABLED"
                }
            },
    "deploymentConfiguration": {
                "maximumPercent": 200,
                "minimumHealthyPercent": 100
            },
    "desiredCount": 2,
    "healthCheckGracePeriodSeconds": 0,
    "schedulingStrategy": "REPLICA",
    "enableECSManagedTags": false,
    "propagateTags": "NONE"
    
}
EOF

And create the ECS Service, using the registered task definition and the Application Load Balancer.

envsubst <ecs_service_def.template>ecs_service_def.json

##create service 
aws ecs create-service --cluster $cluster \
--service-name ${service_name}-service \
--cli-input-json file://ecs_service_def.json \
--region $region

Verify the service in the AWS Management Console.

Setup DNS

Let’s now setup a Route 53 record set on the domain I host on Route 53. I have created a hosted zone. I will create and point a record set to the ALB I created earlier. I have setup the Common Name (CN) on the cert with this domain [ CN= ecs-end-end-encryption.awsblogs.info ]

Test

Let’s test the TLS handshake with the application using a curl command.

echo quit | openssl s_client -showcerts -servername ecs-encryption.awsblogs.info -connect ecs-encryption.awsblogs.info:443 > cacert.pem

Output:

depth=0 C = US, ST = VA, L = Richmond, O = awsblogs.info, OU = awsblogs.info, CN = ecs-encryption.awsblogs.info, emailAddress = user@email.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = VA, L = Richmond, O = awsblogs.info, OU = awsblogs.info, CN = ecs-encryption.awsblogs.info, emailAddress = user@email.com
verify return:1
DONE

##Hit the service 
curl --cacert cacert.pem https://ecs-encryption.awsblogs.info/service

Output:

Hello from behind Envoy proxy!!

There you have it. We encrypted traffic from client to ALB and from ALB to the application (through Envoy front proxy) and we didn’t have to modify the application.

By the way, Envoy has many other uses and many popular Service Meshes use Envoy for data plane. Just like I transparently added the encryption to my application, I could also use Envoy to generate access logs, traces and network metrics. Instead of managing and configuring individual Envoy proxies, Service Meshes give you centralized management of service proxies. Learn more about App Mesh here.

Resources

AWS Cloud Formation templates for ECS infrastructure setup
Maintaining Transport Layer Security
Encryption end-end using NLB
Encryption All The Way To The Container In ECS With Envoy