Why can't my AWS Load Balancer Controller find my subnet in Amazon EKS?

Last updated: 2021-10-06

My AWS Load Balancer Controller can't find my subnet in Amazon Elastic Kubernetes Service (Amazon EKS).

Short description

You receive an error if your AWS Load Balancer Controller can't find your subnet in Amazon EKS.

If you receive the following error, then your service account's AWS Identity and Access Management (IAM) role for the AWS Load Balancer Controller doesn't have the required permissions:

{"level":"error","ts":1621443417.9175518,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":" ingress-2048","namespace":" game-2048","error":"couldn't auto-discover subnets: UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 72ee57ae-f804-4f81-b069-8b04114b67b0"}

To resolve the preceding error, complete the steps in the Resolve the permission denied error section:

-or-

If you receive the following error, then your AWS Load Balancer Controller can't discover at least one subnet:

{"level":"error","ts":1608229710.3212903,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"ingress-2048","namespace":"game-2048","error":"couldn't auto-discover subnets: unable to discover at least one subnet"}

To resolve the preceding error, complete the steps in the Resolve the single subnet discovery error section:

-or-

If you receive either of the following errors, then your AWS Load Balancer Controller can't discover two or more qualified subnets.

"msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to resolve 2 qualified subnet with at least 8 free IP Addresses for ALB
{"level":"error","ts":1606329481.2930484,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"reciter-ing","namespace":"reciter","error":"InvalidSubnet: Not enough IP space available in subnet-xxxxxxxxxxxxxx. ELB requires at least 8 free IP addresses in each subnet.\n\tstatus code: 400, request id: 2a37780c-f411-xxxxx-xxxxx-xxxxxxxxx"}

To resolve the preceding errors, complete the steps in the Resolve multiple subnet discovery errors section:

Resolution

Resolve the permission denied error

1.    Verify that your service account is associated with the AWS Load Balancer Controller:

$ kubectl get deploy aws-load-balancer-controller -n kube-system -o yaml | grep -i serviceAccount

Output:

serviceAccount: aws-load-balancer-controller
serviceAccountName: aws-load-balancer-controller

Note: If your deployment is deployed in a different namespace, then replace -n kube-system with the appropriate namespace.

2.    See what IAM role is attached to the service account associated with the AWS Load Balancer Controller:

$ kubectl describe sa aws-load-balancer-controller -n kube-system | grep role-arn

Output:

annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/eksctl-cluster18-addon-iamserviceaccount-kub-Role1-xxxxxxxxxxxxx

3.    Grant ec2:DescribeAvailabilityZones permissions to the IAM role that you identify in step 2.

Resolve the single subnet discovery error

1.    Add the appropriate tags on your subnets to allow the AWS Load Balancer Ingress Controller to create a load balancer using auto-discovery.

Example of private subnets tags:

kubernetes.io/role/internal-elb                Set to 1 or empty tag value for internal load balancers

Example of public subnets tags:

kubernetes.io/role/elb                         Set to 1 or empty tag value for internet-facing load balancers

Note: You can manually assign subnets to your load balancer using the alb.ingress.kubernetes.io/subnets annotation. For more information, see Ingress annotations on the AWS Load Balancer Controller website.

Example of a subnet with the correct tags for a cluster with an internal load balancer (private subnet):

kubernetes.io/role/internal-elb          1

Example of a subnet with the correct tags for a cluster with a public load balancer (public subnet):

kubernetes.io/role/elb

2.    Tag your subnets with the appropriate format.

Key: kubernetes.io/cluster/your-cluster-name

Value: shared or owned

Important: If you're using the AWS Load Balancer Controller version v2.1.1 or earlier, then you must tag your subnets in the preceding format. Tagging is optional for versions 2.1.2 or later. It's a best practice to tag a subnet if any of the following is true:

  • You have multiple clusters that are running in the same VPC.
  • You have multiple AWS services that share subnets in a VPC.
  • You want more control over where load balancers are provisioned for each cluster.

Resolve multiple subnet discovery errors

1.    Confirm that you have at least two subnets in two different Availability Zones, which is a requirement for creating an Application Load Balancer.

Note: You can create a Network Load Balancer with a single subnet.

2.    For each subnet, specify a CIDR block with at least a /27 bitmask (for example: 10.0.0.0/27) and at least eight free IP addresses.

3.    Confirm that the tags on the subnets are formatted correctly. For example, the tags must not have any leading or trailing spaces.


Did this article help?


Do you need billing or technical support?