The Journey to IPv6 on Amazon EKS: Foundation (Part 1)

Introduction

Scaling Kubernetes networking is key to addressing the growth of services and future-proofing infrastructure as the digital landscape continues to evolve. The need for a unique IP address per pod intersects with the challenges of limited IPv4 address space. The finite pool of available IPv4 addresses often forces Kubernetes cluster administrators to use alternatives such as Network Address Translation (NAT) or overlay networking. This practice often leads to performance bottlenecks and increased management overhead. Using IPv6 address space is a more elegant way to manage large-scale Kubernetes clusters. IPv6 provides a significantly larger total IP address space that allows cluster administrators scale applications without devoting efforts towards working around IPv4 limits.

As more organizations continue to containerize their workloads, they are evaluating the adoption of IPv6. Public IPv4 space exhaustion is often perceived as the major driver. After speaking to some of these customers, we learned that the limited IPv4 Private IP space (i.e., RFC1918) is very often the dominant driver.

The challenge

Adoption of IPv6 can help simplify network configuration and help operate at scale. However, many EKS customers still see the journey to IPv6 adoption as complex. Interviewing some of them, we learned about two key concerns:

• IPv6 Support for AWS networking services
• Amazon Elastic Kubernetes Service (Amazon EKS) interoperability with IPv6 and IPv4 (Hybrid mode)

In this blog series, we will try to demystify the key concerns and provide guidelines, by referencing architectures that are focused on helping customers migrate Amazon EKS clusters to the IPv6 address family. It is beyond this series to dive deeper into IPv6 on AWS. For an IPv6 on AWS holistic view (i.e., planning, architecture, and AWS services) we recommend you that read this whitepaper: IPv6 on AWS. In this part of the series, we will focus on the IPv6 foundation on which Amazon EKS is built.

Kubernetes and IPv6

The upstream Kubernetes approach for IPv6 is dual-stack networking for Pods and services.

Essentially, each Pod and service can be assigned a routable IPv4 and IPv6 address. This is referred to as dual-stack pod and service.

The main advantage of this approach is that Pods can connect to IPv4 or IPv6 endpoints without the dependency on constructs, such as NAT64 or DNS64, or without need for a pass-through interoperability layer.

Consequently, this dual-stack approach means that each Pod consumes a valid IPv6 and private IPv4 address from its associated Amazon Virtual Private Cloud (Amazon VPC) that underlies the dual-stack subnet. Essentially, this means that we are somehow back to square one, facing Private IPv4 exhaustion and network overlapping challenges.

Amazon EKS took a different approach to support IPv6, and it doesn’t support the upstream Kubernetes dual-stack approach yet.

Amazon EKS and IPv6

Working backward from our customers’ requirements, the Amazon EKS team has built an opinionated mechanism to support Amazon EKS IPv6 based clusters adhering to the following five tenets:

1. Allow Pod to Pod/Service in-cluster IPv6 native communication.
2. Allow IPv6 Pod to seamlessly (without the overhead of NAT64/DNS64) connect to an IPv4 endpoint (cross-clusters/VPCs, internet, in-VPC, in-Region).
3. IPv6 Pods connecting to IPv4 endpoints don’t consume routable private IPv4 address from the underlying VPC subnets, hence mitigating the Private IPv4 address space exhaustion issue.
4. Allow IPv4 Amazon EKS clusters to co-exist with Amazon EKS IPv6 clusters. So, Pods in IPv4 Amazon EKS clusters can seamlessly connect to Pods in IPv6 Amazon EKS clusters, without IPv4 to IPv6 external constructs.
5. Allow IPv4 and IPv6 endpoints to connect to load balanced Amazon EKS IPv6 Pods, enabling seamless IPv4 to IPv6 ingress.

The above tenets are implemented by deploying an EKS cluster in IPv6 mode using the corresponding Amazon VPC CNI plug-in, which deeply integrates with the Amazon VPC dual-stack foundation. For ingress connectivity (i.e., tenet 5), the Amazon EKS load balancer controller (LBC) is used to consume service/ingress specific annotations, which results in a dual-stack Network Load Balancer (NLB) or Application Load Balancer (ALB).

Let’s dive deep into the actual architecture based on those tenets.

Solution overview

Step 1: The foundation

In AWS, there are three known networking IP modes that the Elastic network interfaces (ENI) may operate: IPv4-only, IPv6-only and dual-stack. First, you need to create a dual stack VPC. In dual-stack mode, resources can communicate over both IPv4 and IPv6. A separate interoperability layer isn’t required.

IPv6-based Amazon EKS clusters only support a dual-stack VPC foundation. In dual-stack VPC mode, the VPC-CNI is the interoperability layer.

The following diagram depicts a dual-stack VPC foundation:

Figure 1: dual-stack VPC foundation as a mandatory base for Amazon EKS IPv6 based clusters

A dual-stack VPC supports both IPv4 and IPv6 address ranges, known as CIDRs (Classless Inter-Domain Routing), within its subnets. The subnets are created with associated CIDR blocks align with the allocated IPv4 and IPv6 ranges, ensuring that resources can communicate using both address types within the same VPC infrastructure. When an Amazon Elastic Compute Cloud (Amazon EC2) instance is deployed into a dual-stack subnet the Amazon EC2 ENI will consume a single IPv4 and an IPv6 address, which essentially is called a dual-stack Amazon EC2 instance.

Step 2: Amazon EKS IPv6-based cluster

Once Step 1 (i.e., VPC foundation) is deployed as depicted previously (Figure 1), we can proceed to deploying an Amazon EKS IPv6 cluster. So, what does this Amazon EKS in IPv6 mode mean?

Amazon EKS in IPv6 mode simply means:

• The Amazon EKS control plane is created with, –ip-family ipv6 API option. In addition to the Amazon EKS control plane, normal bootstrap operations in the Amazon EKS service will allocate an immutable Unique Local IPv6 Unicast Address (ULA). This IPv6 range is non-publicly-routable and will be used to assign IPv6 addresses for Kubernetes (i.e., ClusterIP) based services, which are IPv6 only (see 1 in Figure 2).
• When Amazon EKS worker nodes are bootstrapped into dual-stack subnets (see 2 in Figure 2) as part of the Amazon EKS data-plane, by default, the Amazon VPC CNI will attempt to carve and attach a /80 prefix. This can yield around 10^14 IPv6 addresses for Pods — IP exhaustion solved!

Any Pod that will land on this dual-stack Amazon EC2 worker-node will be assigned an IPv6 address allocated from the /80 prefix. (See steps A to C on left side of Figure 2).

Figure 2: Amazon EKS IPv6 Cluster on top of the mandatory dual-stack VPC foundation

Here, it is important to note that all the other Amazon EC2 and VPC constructs, such as Amazon EC2 Security groups, routing tables, Network Access Control Lists (NACLs) support IPv6. You can use Amazon EC2 security groups to define rules that allow inbound and outbound network traffic to and from IPv6 Pods directly.

Besides, there is a key consideration when it comes to setting up IPv6 egress to the internet. IPv6 addresses are publicly routable. Therefore, for IPv6 Pods to egress the internet it is recommended to create an Egress-Only-Internet-Gateway (EIGW) and attach it to the dual-stack VPC, and then configure the appropriate IPv6 default route at the subnet routing table (Figure 1). The NAT Gateway is still used to publicly allow private IPv4 endpoints to connect to the internet.

Once Step 1 and Step 2 are completed and an IPv6-based Amazon EKS cluster is configured, the tenets are now in-action. Give it a try and deploy an Amazon EKS cluster with our EKS terraform based EKS/IPv6 blueprint.

Conclusion

In this post, we covered an overview of the foundational aspects of an Amazon EKS IPv6 cluster. We explained how Amazon EKS helps our customers build scalable clusters using IPv6, solving the limited IPv4 address space challenge. Amazon EKS in IPv6 helps reduce operational overhead, mitigating the need for unwarranted interoperability layers from the beginning and allowing communication with any IPv4 endpoint as well as AWS services that support IPv6. We strongly recommend you start adopting Amazon EKS/IPv6 today rather than waiting for your entire organizational IPv6 migration to materialize. In the next two posts, we will dive deep into common implementation patterns and migration strategies.