How can I tag the Amazon VPC subnets in my Amazon EKS cluster for automatic subnet discovery by load balancers or ingress controllers?

Last updated: 2020-02-07

I want to deploy load balancers or ingress controllers in the public or private subnets of my Amazon Virtual Private Cloud (Amazon VPC). Why can't my subnets be discovered by Kubernetes in my Amazon Elastic Kubernetes Service (Amazon EKS) cluster?

Short Description

To identify a cluster's subnets, the Kubernetes Cloud Controller Manager (cloud-controller-manager) and ALB Ingress Controller (alb-ingress-controller) query that cluster's subnets by using the following tag as a filter:

kubernetes.io/cluster/cluster-name

The Cloud Controller Manager and ALB Ingress Controller both require subnets to have either of the following tags:

kubernetes.io/role/elb

--or--

kubernetes.io/role/internal-elb

Note: If you don't use the preceding tags, then Cloud Controller Manager determines if a subnet is public or private by examining the route table associated with that subnet. Unlike private subnets, public subnets use an internet gateway to get a direct route to the internet.

If you don't associate your subnets with either tag, then you receive an error.

For example, if you're troubleshooting the Kubernetes service and you run the kubectl describe service your-service-name command, then you receive the following error:

Events:
  Type     Reason                      Age               From                Message
  ----     ------                      ----              ----                -------
  Normal   EnsuringLoadBalancer        9s (x2 over 14s)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  9s (x2 over 14s)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service default/guestbook: could not find any suitable subnets for creating the ELB

If you're troubleshooting the ALB Ingress Controller and you run the kubectl logs your-alb-ingress-controller-pod-name command, then you receive the following error:

E0121 22:44:02.864753       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to
retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the kubernetes.io/cluster/\u003ccluster name\u003e tag with a value of shared or owned and the kubernetes.io/role/elb tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"2048-ingress"}

Note: If you create the VPC using eksctl, then all the subnets in that VPC have the kubernetes.io/role/elb and kubernetes.io/role/internal-elb tags.

Resolution

Choose the appropriate option for tagging your subnets:

For public and private subnets used by load balancer resources

Tag all public and private subnets that your cluster uses for load balancer resources with the following key-value pair:

Key: kubernetes.io/cluster/cluster-name
Value: shared

The cluster-name value is for your Amazon EKS cluster. The shared value allows more than one cluster to use the subnet.

For private subnets used by internal load balancers

To allow Kubernetes to use your private subnets for internal load balancers, tag all private subnets in your VPC with the following key-value pair:

Key: kubernetes.io/role/internal-elb
Value: 1

For public subnets used by external load balancers

To allow Kubernetes to use only tagged subnets for external load balancers, tag all public subnets in your VPC with the following key-value pair:

Key: kubernetes.io/role/elb
Value: 1

Note: Use the preceding tag instead of using a public subnet in each Availability Zone.