Serve distinct domains with TLS powered by ACM on Amazon EKS
AWS Elastic Load Balancers provide native ingress solutions for workloads deployed on Amazon Elastic Kubernetes Service (Amazon EKS) clusters at both L4 and L7 with Network Load Balancer and Application Load Balancer (ALB). The AWS Load Balancer Controller, formerly called the AWS ALB Ingress Controller, satisfies Kubernetes ingress using ALB and service type load balancing with Network Load Balancer (NLB). When a Kubernetes ingress is created, an ALB is provisioned that load balances HTTP or HTTPS traffic to different nodes or pods within the cluster. By default, the Load Balancer Controller creates a new ALB for each ingress resource that matches its requirements via specific Kubernetes annotations.
When an ingress resource with the annotation kubernetes.io/ingress.class: alb is detected by the AWS Load Balancer Controller, it creates various AWS resources based on the configuration specified in the ingress resource as annotations, including ALB, TargetGroups, Listeners, and Rules. A few noteworthy annotations from the full list available here include the alb.ingress.kubernetes.io/scheme that creates an internet-facing or internal ALB, alb.ingress.kubernetes.io/subnets annotation that specifies a list of target subnets for the ALB, and alb.ingress.kubernetes.io/certificate-arn annotation for specifying one or more certificates managed by AWS Certificate Manager, which enables HTTPS traffic on the ALB. This annotation-based configuration provides a flexible and powerful mechanism to provision the desired AWS resources.
Many organizations have a need to serve hundreds of distinct domain names with Transport Layer Security (TLS) e.g., a web hosting provider. Having a dedicated load balancer per application can become costly and cumbersome to manage.
The AWS Load Balancer Controller solves this challenge by serving traffic to distinct domains by using ALB’s host-based routing, paired with Server Name Indication (SNI). Utilizing the annotation-based configuration, this post shows how a single ALB is provisioned and configured to securely serve three demonstration websites that’re aimed at serving pet lovers of rabbits, chipmunks, and hamsters.
Below is a high-level illustration on how the traffic will be served to different applications using a single application load balancer.
In the diagram above:
- A user issues a GET request to https://hamster.local/
- The ALB receives the request, terminates the TLS, and using SNI extracts the host header
- Then using the host-named based routing the ALB maps the request to the proper target group
- Target group maps to the corresponding application Pods for your service
High-level the steps of the solution:
- Deploy AWS Load Balancer Controller
- Setup and import TLS certificates into AWS Certificate Manager (ACM)
- Deploy three distinct applications
- View Application Load Balancer SNI rules
- Create Route 53 availability zones and records
- Validate Kubernetes resources
- Validate the solution
Note: Currently, the ALB has a default quota of up to 25 certificates per ALB.
You’ll need these tools to follow the walkthrough:
Step A: Deploy AWS Load Balancer Controller
In this step, we’ll create and setup the AWS Load Balancer Controller. For in depth granular instructions, please browse this link.
Replace the example values with your own values.
Fetch the AWS IAM policy:
Create an AWS IAM policy using the policy downloaded in the previous step:
Create an AWS IAM role:
Replace MY-EKS-CLUSTER-NAME with the name of your Amazon EKS cluster, 111122223333 with your account ID, and then run the command.
Using Helm (v3), Install the Amazon Load Balancer Controller:
Replace MY-EKS-CLUSTER-NAME with the name of your Amazon EKS cluster:
Verify that the Amazon Load Balance Controller is installed:
Step B: Setup TLS certificates for ACM
We’ll be creating TLS certificates for domains rabbit.local, hamster.local, and chipmunk.local and upload them and their private keys into the AWS Certificate Manager.
Create the TLS certificate for rabbit.local:
Note: -nodes flag used in the openssl command is passed to prevent the encryption of the private key for simplicity purposes.
Upload certificate to ACM
Repeat the same process for hamster.local and chipmunk.local domains, respectively.
Step C: Deploy three distinct applications into the Amazon EKS cluster
Now that the edge is deployed, let’s deploy the entire Kubernetes application backend and its resources. Once deployed we’ll wire the edge to the proper applications.
To provision the necessary resources, the following manifest creates the following:
- An nginx server Pod, hosting the landing HTML page
- A configmap that holds the nginx configuration for the static HTML content
- Kubernetes Service object to expose the pod as Kubernetes service
- An Ingress object providing HTTPS access to the service via the ALB
By default, satisfying an ingress result in the creation of an ALB per ingress. In our case, we’re looking to consolidate multiple ingress resources into a single ALB (i.e., each ingress represents the distinct domain name and points to the same ALB and SNI certificate).
To achieve this, we set the ingress annotation:
Note: IngressGroup Security Best Practice state that the IngressGroup feature should be used when all the Kubernetes users are within a trusted boundary. See this link for more details.
Creating the Template-based manifest:
Apply the manifests:
Validate that hamster, chipmunk, rabbit variables were substituted correctly:
Step D: View Application Load Balancer SNI rules
Head over to the AWS Management Console and select the Name of the load balancer.
Navigate to the Rules to view the listener rules. We can see the three rules created for our applications pointing to their associated target group, as shown in the following diagram.
Step E: Create Route53 Zones and records
For demonstration purposes, we’ll create three private hosted zones in Route53 to serve traffic in a private setup. In the real world, it is highly likely that you’ll create public hosted zones, which point to the public facing application load balancer.
Create three private hosted zone on Route53 using the AWS CLI:
Input the VPC-ID (the one that the Amazon EKS cluster is deployed into.
Repeat the command above to create private zones for chipmunk.local and hamster.local domains.
Validate that all three zones were created:
Pointing the domain to the Application Load Balancer:
Replace the Private-hosted-zone-id with yours, replace the DNS-of-ALB with the application load balancer created in Step C.
Repeat the prior step for hamster.local and chipmunk.local domains.
Step F: Validate Kubernetes resources
Validate the Kubernetes resources:
Step G: Validating the solution
With the applications deployed and all relevant resources up and running, let’s test our applications.
- Use System Manager to connect to one of you Amazon EKS Worker nodes
- Execute the following commands:
To avoid incurring future charges, delete the application resources:
Uninstall the AWS load Balancer Controller:
Delete the service account for AWS Load Balancer Controller. Replace MY-EKS-CLUSTER-NAME with the name of your cluster:
Delete the AWS IAM Policy for the AWS Load Balancer Controller. Replace 111122223333 with your account ID.
In this post, we showed you how to serve multiple domains by using a single instance of an AWS Application Load Balancer using host-based routing with SNI. We created an Ingress resource with annotations of IngressGroup, TLS (via ACM), and traffic listener to serve HTTPS traffic to distinct domains by a single ALB. This solution reduces the management overhead and is a cost-effective option as it doesn’t require creating dedicated ALB per domain.