Containers

Setting up end-to-end TLS encryption on Amazon EKS with the new AWS Load Balancer Controller

In this blog post, I’ll show you how to set up end-to-end encryption on Amazon Elastic Kubernetes Service(Amazon EKS). End-to end encryption in this case refers to traffic that originates from your client and terminates at an NGINX server running inside a sample app.

I work with regulated customers who need to satisfy regulatory requirements like PCI DSS, HIPAA, and so on. Until recently, I didn’t have simple and effective solution to propose to them. Although it was possible to use ingress controllers like the NGINX Ingress Controller or Traefik fronted by a Network Load Balancer, configuring end-to-end encryption was cumbersome and difficult to automate.

The new AWS Load Balancer manages AWS Elastic Load Balancers for a Kubernetes cluster.  The controller provisions an AWS Application Load Balancer (ALB) when you create a Kubernetes Ingress and an AWS Network Load Balancer (NLB) when you create a Kubernetes Service of type LoadBalancer using IP targets on 1.18 or later Amazon EKS clusters.

TLS encryption of ingress traffic to Amazon EKS

One of the ways to intelligently route traffic that originates outside of a cluster to services running inside the cluster is to use Ingress controllers. They are usually fronted by a layer 4 load balancer like the Classic Load Balancer or the Network Load Balancer. The following diagram shows the places in a network where encrypted traffic can be terminated:

 

1.     Terminate traffic at the load balancer.
2.     Terminate traffic at the ingress.
3.     Terminate traffic on the pod.

How and where you terminate your TLS connection depends on your use case, security policies, and need to comply with various regulatory requirements.

If there is no strict requirement for end-to-end encryption, try to offload this processing to the Ingress Controller or the NLB. This will help you to optimize the performance of your workloads and make them easier to configure and manage.

In this post, I use a scenario where there is a requirement to have end-to-end TLS encryption and preserve the client IP address. The new AWS Load Balancer Controller supports a Network Load Balancer (NLB) with IP targets for pods running on Amazon EC2 instances and AWS Fargate through Kubernetes service of type LoadBalancer with proper annotation. In this mode, the AWS NLB targets traffic directly to the Kubernetes pods behind the service, eliminating the need for an extra network hop through the worker nodes in the Kubernetes cluster, which decreases latency and improves scalability. Smaller target groups reduces management complexity and makes it less likely you will hit NLB limits in large clusters.

Prerequisites

To complete this tutorial, you need the following:

  • An AWS account.
  • The AWS Command Line Interface (AWS CLI), with the kubectl and eksctl tools installed and configured. Follow the instructions in Getting started with Amazon EKS – eksctl in the Amazon EKS User Guide.
  • Helm CLI.
  • The openssl program, a command line tool for using the various cryptography functions of OpenSSL’s crypto library from the shell.

If you already have these tools installed, be sure to update them before you begin.

Provision an Amazon EKS cluster

You can use the AWS Management Console or AWS CLI, but I recommend using eksctl to provision the cluster. Change node-type and region as appropriate for your environment. Cluster provisioning takes approximately 15 minutes.

eksctl create cluster \
--name nlb-lab \
--version 1.18 \
--nodegroup-name nlb-lab-workers \
--node-type t2.medium \
--nodes 2 \
--region us-west-2

This eksctl command creates an Amazon EKS cluster in the us-west-2 Region with Kubernetes version 1.18 and two nodes.

I use the t2.medium instance family in this example. If you’re creating an Amazon EKS cluster in your production environment, use the instance family type appropriate for your needs.

Use the following command to verify the cluster is running and kubectl is properly configured:

kubectl get nodes

Your output should look like this:

NAME                                             STATUS    ROLES      AGE    VERSION
ip-192-168-39-68.us-west-2.compute.internal      Ready     <none>     71s    v1.18.8-eks-7c9bda
ip-192-168-81-36.us-west-2.compute.internal      Ready     <none>     77s    v1.18.8-eks-7c9bda

Install AWS Load Balancer Controller

Follow the steps in AWS Load Balancer Controller Installation. You can use Helm or YAML manifests.

Use the following command to verify that AWS Load Balancer Controller is running:

kubectl get pods -n kube-system

You should see the aws-load-balancer-controller pod is ready with a status of Running:

 

Generate a TLS certificate for a backend application

If you own a custom domain, you can sign certificates using certbot and then create a DNS record that points to the provisioned NLB DNS name. Otherwise, you can use SSL with a self-signed certificate for development and testing.

Use the following command to generate a self-signed certificate:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nlb-lab.com"

Now generate a secret in the Amazon EKS cluster, so your test pod can use the certificate:

kubectl create secret tls nlb-tls-app-secret --key tls.key --cert tls.crt

Deploy the demo application and service

Now deploy the demo application and service to your Amazon EKS cluster. The demo application is a simple NGINX web server configured to return Hello from pod hostname.  You can take below complete YAML, and then save it to a file named nlb-tls-app.yaml and apply it to your cluster using following command:

kubectl apply -f nlb-tls-app.yaml

Before you run the command, these are the important parts of the configuration and the changes you need to apply.

In the Service object, there are two annotations:

  •  aws-load-balancer-type instructs AWS Load Balancer Controller to provision the Network Load Balancer for your application.
  • aws-load-balancer-proxy-protocol enables proxy protocol version 2 on the Network Load Balancer, which allows the client source IP addresses to be passed to your pod.
annotations:
  service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

The ConfigMap object contains the following configuration for the NGINX server.  Use your Amazon EKS cluster VPC CIDR range in the set_real_ip_from directive.  Use your domain name, or if you are using a self-signed certificate, use the DNS name of the Network Load Balancer in server_name directive.

app.conf: |-
    server {
      # Instruct nginx server to listen on port 8443 using SSL protocol
      # and accept proxy protocol 
      listen 8443 ssl proxy_protocol; 
      # Get client source IP from Proxy protocol header
      real_ip_header proxy_protocol;
      # Here you need to set your EKS cluster VPC CIDR range
      set_real_ip_from 192.168.0.0/16;
      # Here you need to put your domain name or if you have used self-signed
      # certificates, DNS name of NLB
      server_name your_server_name;

Note: If you are using a self-signed certificate, you will not know the NLB DNS name until you deploy the application. After you deploy it, go to the AWS console , copy the NLB DNS name, and then run the following command to edit ConfigMap and update server_name with the NLB DNS name.

kubectl edit configmap secure-config

Complete YAML manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nlb-tls-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nlb-tls-app
  template:
    metadata:
      labels:
        app: nlb-tls-app
    spec:
      containers:
        - name: nlb-tls-app
          image: nginxdemos/nginx-hello:plain-text
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: secret
              mountPath: /etc/nginx/ssl
              readOnly: true
            - name: config-volume
              mountPath: /etc/nginx/conf.d
      volumes:
        - name: secret
          secret:
            secretName: nlb-tls-app-secret
        - name: config-volume
          configMap:
            name: secure-config
---
apiVersion: v1
kind: Service
metadata:
  name: nlb-tls-app
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
spec:
  ports:
    - port: 443
      targetPort: 8443
      protocol: TCP
      name: https
  selector:
    app: nlb-tls-app
  type: LoadBalancer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-config
data:
  app.conf: |-
    server {
      listen 8443 ssl proxy_protocol;
      real_ip_header proxy_protocol;
      set_real_ip_from 192.168.0.0/16;
      server_name <your_server_name>;

      ssl_certificate /etc/nginx/ssl/tls.crt;
      ssl_certificate_key /etc/nginx/ssl/tls.key;

      default_type text/plain;

      location / {
        return 200 "hello from pod $hostname\n";
      }
    }

 

After you have applied manifest to the cluster, run the following command to verify that the application is up and running:

kubectl get pods

You should see that the nlb-test-app pod is running with a status of Ready.

 

Verify end-to-end encryption

Now use the openssl command to verify end-to-end TLS encryption. Use your custom domain name or NLB DNS name, as appropriate.

openssl s_client -connect <your_server_name>:443

You should see a successful TLS handshake and other details in the output:

CONNECTED(00000005)
depth=0 CN = nlb-lab.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = nlb-lab.com
verify return:1
---
Certificate chain
 0 s:/CN=nlb-lab.com
   i:/CN=nlb-lab.com
---
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---5t
SSL handshake has read 2089 bytes and written 289 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 3668A8CD5BE7CD528570F876A5D7F32C48825684B3860A7A2A98E0D37B6C7BA5
    Session-ID-ctx:
    Master-Key: 31E909FD4A205E269921BDAF07C98372EB0A0326ACC1729ACF159213527024ADE0BB905F5AB74D2854D932BE8FBBBF00
    TLS session ticket lifetime hint: 300 (seconds)

Now you can verify that the client source IP address is preserved.  In your browser, visit https://<your_server_name>, and then run the following command. Use the name of your pod to view pod logs and check that requests are coming from your IP address.
Bash
kubectl logs nlb-tls-app-57b67f67f-nmqj9

Example output:
xxx.xxx.xxx.xxx – – [14/Nov/2020:00:09:47 +0000] “GET / HTTP/1.1” 200 43 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0” “-”
xxx.xxx.xxx.xxx – – [14/Nov/2020:00:09:47 +0000] “GET /favicon.ico HTTP/1.1” 200 43 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0” “-”

Cleaning up

To avoid incurring future charges, perform these cleanup steps.

Delete the application:

kubectl delete -f nlb-tls-app.yaml

Delete the Amazon EKS cluster:

eksctl delete cluster --name=nlb-lab

Conclusion

In this blog post, I showed you how to set up end-to-end TLS traffic encryption to an Amazon EKS cluster using AWS Load Balancer Controller with Network Load Balancer in IP mode. If you have questions or want to contribute, join our aws-load-balancer-controller project on GitHub.