Enabling mTLS with ALB in Amazon EKS

Introduction

In today’s interconnected world, communication faces evolving security threats. From sensitive financial transactions in online banking to secure data transmissions in the automobile industry, ensuring trust and authenticity between businesses is becoming more and more critical. This is where Mutual Transport Layer Security (mTLS) can be an option to offer enhanced security through advanced cryptographic handshake protocols.

While traditional TLS encrypts data in transit, mTLS verifies the identities of both the client and server before establishing a connection, utilizing digital certificates issued by a trusted authority. This two-way authentication creates a mutual trust relationship, guaranteeing that only authorized parties can exchange information.

For businesses dealing with complex applications, managing mTLS infrastructure can be daunting. Manually building and maintaining custom solutions often involve substantial developer resources, updates, and infrastructure investments. Moreover, self-created or third-party solutions can increase time and management overhead, diverting focus from core business objectives.

TLS Connection Workflow

Figure 1: TLS connection workflow

mTLS Connection Workflow

Figure 2: mTLS connection workflow

To help our customers improve their security posture without additional management overhead, in 2023 we launched native support for mTLS within Application Load Balancer (ALB), which eliminated the need for such complex custom solutions by allowing you to offload client authentication to the load balancer itself. As a result, developers gained valuable time and resources to focus on application developments, while enjoying a fully managed, scalable, and cost-effective solution for securing their environment.

In this post, we will show you how you can seamlessly use ALB’s native mTLS capabilities with Amazon Elastic Kubernetes Service (Amazon EKS) workloads, making use of a recently released features of the AWS Load Balancer Controller (version 2.7) that added support for (mTLS) Mutual Transport Layer Security on Kubernetes Ingress.

Solution overview

This post is dedicated to ensuring secure communications between Kubernetes workloads with mTLS in Amazon EKS. It covers the following components:

AWS Load Balancer Controller: A Kubernetes controller to help manage Elastic Load Balancers for the Kubernetes cluster
ExternalDNS: A Kubernetes component that allows for automatically creating and managing DNS records for services exposed externally with supported DNS providers such as Amazon Route 53. It facilitates external communication with services running inside the Kubernetes cluster by resolving the service’s hostname to the external IP address of the Kubernetes cluster.
Sample Application Deployment: Deploy a sample workload for an mTLS-enabled service. This workload encompasses the deployment of a sample application configured for mutual TLS (mTLS) within a Kubernetes environment, with a specific focus on Amazon EKS.

mTLS support for Kubernetes Ingress in Amazon EKS

Figure 3: The red lines indicates the implementation of mTLS with both client and server presenting their certificate to each other and using a private certificate authority (CA) to authenticate clients before granting access.

Prerequisites

You have an Amazon EKS cluster, otherwise create a new cluster.
You have properly installed and configured latest versions of AWS Command Line Interface (AWS Command Line Interface [AWS CLI]), kubectl, and Helm CLI
A custom domain name to test your sample application
A certificate managed by AWS Certificate Manager

Walkthrough

Step 1: Make a self-signed certificate

To test the mutual authentication on Application Load Balancer, follow the step-by-step instructions below to make a self-signed CA bundle and client certificate with x509v3 extensions using OpenSSL:

Create the private certificate authority (CA) private and public keys:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My BuildOn AWS Cert Authority'

Prepare X.509 Extensions configuration file. Save the content below into a file named custom_openssl.cnf

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth

Generate the Client Key, and Certificate and Sign with the CA Certificate:

openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My BuildOn AWS mTLS Client'
openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -out client.crt -CAcreateserial -CAserial serial -extfile custom_openssl.cnf

For a production environment, you can implement mutual authentication on ALB using a CA bundle with roots and/or intermediate certificates generated by AWS Private Certificate Authority as the source of trust to validate your client certificates. Follow the step-by-step instructions to make a CA bundle and client certificate using AWS Private Certificate Authority.

Step 2: Create a trust store

To create a trust store, we will use the CA certificate as the certificate bundle. This will require creating an Amazon Simple Storage Service (Amazon S3) bucket, then uploading your CA bundle to the Amazon S3 bucket. You complete this task with the following commands:

aws s3 mb s3://somerandom-trsut-store --region us-east-2

aws s3api put-bucket-versioning --bucket somerandom-trsut-store --versioning-configuration Status=Enabled

aws s3 cp ca.crt s3://somerandom-trsut-store/trust-store/  --region us-east-2

aws elbv2 create-trust-store --name pca-certs \
    --ca-certificates-bundle-s3-bucket somerandom-trsut-store \
    --ca-certificates-bundle-s3-key trust-store/ca.crt  --region us-east-2

Expected result:

View Trust Store created for ALB

Figure 4: Trust Store created for ALB on AWS Console

Step 2: Install ALB Controller

The controller in your cluster needs access to the AWS ALB/NLB APIs with AWS Identity and Access Management (AWS IAM) permissions. Refer to the steps in the documentation to configure the recommended AWS IAM roles for service accounts (IRSA) for the controller. To install Application Load Balancer Controller, you can use Helm. Retrieve the VPC automatically created for the cluster and substitute values below with your own:

export AWS_DEFAULT_REGION="us-east-2"
export vpcid=$(aws eks describe-cluster --name fg-security-quickstart --query 'cluster.resourcesVpcConfig.vpcId' --output text)
export mycluster=fg-security-quickstart
export region="us-east-2"

For mTLS support in ALB provisioned by Kubernetes Ingress resource, you will need to install the version 2.7 or later version of the AWS Load Balancer Controller. Install the AWS Load Balancer Controller with the Helm command below:

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=$mycluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=$region \
  --set vpcId=$vpcid

Step 3: Install and Setup External DNS (Optional)

You may create a DNS record in Amazon Route53 for your application. Refer to the documentation to setup and manage records in Route 53 that point to controller deployed ALBs. Alternatively, you can set up ExternalDNS in your Kubernetes cluster to create DNS record in Amazon Route53 on your behalf, see Setting up ExternalDNS for services on AWS (on the GitHub website) and Set up ExternalDNS. Be sure to use 0.11.0 version or higher of ExternalDNS.

Download sample external-dns

wget https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/external-dns.yaml

Open the downloaded external-dns.yaml file and edit the –domain-filter flag to include your hosted zone(s). The following example is for a hosted zone com:

args:
- --source=service
- --source=ingress
- --domain-filter=example.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
- --provider=aws
- --policy=upsert-only 
- --aws-zone-type=public 
- --registry=txt
- --txt-owner-id=my-identifier  # Your Route53 Zone ID

Deploy the downloaded external-dns.yaml file:

kubectl apply -f external-dns.yaml

Verify it deployed successfully.

kubectl logs -f $(kubectl get po | egrep -o 'external-dns[A-Za-z0-9-]+')

Step 4: Deploy a sample application

Prepare your environment by declaring the variables below:

CERTIFICATE_ARN="YOUR-CERTIFIVATE-ARN" # ARN of certificate managed by AWS Certificate Manager
TRUSTORE_ARN="YOUR-TRUST-STORE-ARN"
SERVICES_DOMAIN="example.com"

Copy and paste the content below into a file named mtls.yaml to create a sample workload:

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: mtls-app
  namespace: mtls  
  labels:
    app: mtls
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mtls
  template:
    metadata:
      labels:
        app: mtls
    spec:
      containers:
      - name: mtls-app
        image: hashicorp/http-echo
        args:
          - "-text=mTLS Sample Application in Amazon EKS"
---
kind: Service
apiVersion: v1
metadata:
  name: mtls-service
  namespace: mtls  
spec:
  selector:
    app: mtls
  ports:
    - port: 80 
      targetPort: 5678
      protocol: TCP
  type: NodePort

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: mtls
  name: mtls-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/load-balancer-name: mtls-demo
    alb.ingress.kubernetes.io/tags: auto-delete=no    
    # Listen port configuration
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 80}, {"HTTPS": 443}, {"HTTPS": 8080}, {"HTTPS": 8443}]'
    ## TLS Settings
    alb.ingress.kubernetes.io/certificate-arn: ${CERTIFICATE_ARN}  # specifies the ARN of one or more certificate managed by AWS Certificate Manager
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06 # Optional (Picks default if not used)
spec:
  ingressClassName: alb
  rules:
    - host: "mtls.${SERVICES_DOMAIN}"
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: mtls-service
              port:
                number: 80

In this example, we specified four HTTPS listeners with ports 80, 443, 8080, and 8443 for the ingress. A new attribute to enable connection logs for your application loadbalancer is also included in the annotation. When you enable connection logs, you must specify an Amazon S3 bucket for the connection logs. For more information, refer to the Amazon S3 bucket requirements here.

Deploy the manifest:

kubectl create namespace mtls

kubectl create -f mtls.yaml

After few minutes, run the following command to verify the ingress created:

kubectl get ingress -n mtls

Expected output:

NAME           CLASS   HOSTS                                  ADDRESS                                                               PORTS   AGE
mtls-ingress   alb     mtls.example.com   k8s-mtls-mtlsingr-a49f674499-1121962520.us-east-2.elb.amazonaws.com                       80      9m43s

Run a curl command to test connectivity to the application:

curl -sk -v https://mtls.example.com

You should see a 200 HTTP response with TLS verification happening once. See CERT verify in the example output:

* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
< HTTP/2 200 
mTLS Sample Application in Amazon EKS

Step 5: Associate a trust store to the Ingress Resource

mTLS for ALBs provides two options for validating X.509v3 client certificates: passthrough and verify mode. For passthrough mode, you can configure the listener to accept any certificate(s) from the client. For verify mode, you will need to create a new Trust Store, upload your CA bundle, and attach the Trust Store to your listener that is configured to verify client certificates.

Edit the ingress annotation section in the previous mtls.yaml file and add the annotation below to associate the trust store created in previous step:

---
kind: Ingress
...
  annotations:
    # mTLS configuration
    alb.ingress.kubernetes.io/mutual-authentication: '[{"port": 80, "mode": "passthrough"}, 
                                        {"port": 443, "mode": "verify", "trustStore": "$TRUSTORE_ARN", "ignoreClientCertificateExpiry" : true}]'
    # New load balancer attributes for enabling connection logs for Ingresses                                        
    alb.ingress.kubernetes.io/load-balancer-attributes: connection_logs.s3.enabled=true,connection_logs.s3.bucket=connection-log-bucket-name,connection_logs.s3.prefix=ingress-mtls-alb
...

In this example, the listener HTTPS:80 will be set to passthrough mode, the listener HTTPS:443 will be set to verify mode and will be associated with the provided trust-store-arn arn:aws:elasticloadbalancing:trustStoreArn. The remaining listeners HTTPS:8080 and HTTPS:8443 will be set to default mTLS mode (i.e., off).

Run the following command to apply the changes from the manifest:

kubectl apply -f mtls.yaml

Verify the ALB created from Amazon Elastic Compute Cloud (Amazon EC2) Console:

View ALB created from AWS EC2 console

Figure 5: ALB details page on Amazon EC2 Console

Run the curl command again with the client certificate and key.

curl https://mtls.example.com --cacert ca.crt --key tls.key --cert tls.crt -v

You should see a 200 HTTP response with TLS verification happening twice. See CERT verify in the example output:

* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, CERT verify (15):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
< HTTP/2 200
mTLS Sample Application in Amazon EKS Fargate

Cleaning up

To avoid incurring future charges, you should delete the resources created during this tutorial. You can delete the resources with the following command:

kubectl delete namespace mtls

eksctl delete cluster -f cluster.yaml

aws elbv2 delete-trust-store --trust-store-arn $truststoreArn --region us-east-2

aws s3api delete-object --bucket somerandom-trsut-store --key trust-store/ca.crt --region us-east-2

aws s3 rb s3://somerandom-trsut-store --force --region us-east-2

aws s3 rb s3://connection-log-bucket-name --force --region us-east-2

Conclusion

In this post, we showed you how to enable mutual TLS at ALB created by a Kubernetes Ingress resource for an application running in Amazon EKS using a self-signed generated certificates. Companies have improved their security posture by using Mutual TLS implementation to add an extra layer of security to their Kubernetes workload traffic and to prevent various kinds of attacks.

Containers