How can I troubleshoot issues when I use the AWS Load Balancer Controller to create a load balancer?

Short description

To troubleshoot load balancer creation issues, complete the following steps:

• Make sure that all prerequisites are met.
• Check the annotations of the Ingress (Application Load Balancer) or Service (Network Load Balancer) object.
• Review the AWS Load Balancer Controller pod's logs for additional information.
• If the cluster runs on AWS Fargate, then verify that a Fargate profile is created for the namespace where the Ingress or Service object is.
• Check for unaddressed dependencies.

The AWS Load Balancer Controller manages Elastic Load Balancing for an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. The controller provisions the following resources:

• An Application Load Balancer when you create a Kubernetes ingress.
• A Network Load Balancer when you create a Kubernetes service of the LoadBalancer type.
  Note: With AWS Load Balancer Controller version 2.3.0 or later, you can create a Network Load Balancer with either the instance or IP target type.

Resolution

Make sure that you meet all prerequisites

For a list of Application Load Balancer prerequisites, see Application load balancing on Amazon EKS. For a list of Network Load Balancer prerequisites, see Network load balancing on Amazon EKS.

1.    Verify that you successfully provisioned the AWS Load Balancer Controller. It's a best practice to use version 2.4.4 or later.

2.    Check the number of subnets. Application Load Balancer must have at least two subnets in different Availability Zones. Network Load Balancer must have at least one subnet. The subnets must have at least eight available IP addresses. For more information, see Create a virtual private cloud (VPC).

3.    You must use the following tag in certain scenarios:

• Key: "kubernetes.io/cluster/cluster-name"
• Value: "shared" or "owned"

For an Application Load Balancer

You must tag one security group in the following scenarios:

• You use multiple security groups that are attached to a worker node.
• You use the AWS Load Balancer Controller version v2.1.1 or earlier.

For a Network Load Balancer

If you use the AWS Load Balancer Controller version v2.1.1 or earlier, then you must tag the subnets.

For information on using the Amazon EC2 console to add tags, see Work with tags using the console. For information on using the AWS Command Line Interface (AWS CLI) to add tags, see Work with tags using the command line.

Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

4.    Unless subnet IDs are explicitly specified as annotations in the Service object or Ingress object, make sure that the subnets have the following tags. Without these tags, Subnet Auto Discovery (on the GitHub website) doesn't work:

Private subnets tag:

• Key: "kubernetes.io/role/internal-elb"
• Value: "1"

Public subnets tag:

• Key: "kubernetes.io/role/elb"
• Value: "1"

Check the annotations of the Ingress or Service object

Verify annotations on the Service object or annotations on the Ingress object.

Note: Other annotations use default values. For a list of all available annotations supported by the AWS Load Balancer Controller for Application Load Balancing, see Ingress annotations on GitHub. For a list of all available annotations supported by the AWS Load Balancer Controller for Network Load Balancing, see Service annotations on GitHub.

Application Load Balancer

• kubernetes.io/ingress.class: alb: IngressClassParams is a custom resource definition (CRD) that's specific to the AWS Load Balancer Controller. You can use this CRD along with the IngressClass parameter field (on the Kubernetes website). Use this parameter to enforce settings for a set of Ingresses.

Network Load Balancer

• With IP targets: Use service.beta.kubernetes.io/aws-load-balancer-type: "external" and service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip".
• With instance targets: Use service.beta.kubernetes.io/aws-load-balancer-type: "external" and service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance".

To view the Service or the Ingress object, run one of the following commands. Replace SERVICE-NAME, INGRESS-NAME, and NAMESPACE with the correct values for your use case:

kubectl describe service SERVICE-NAME -n NAMESPACE  
  
kubectl describe ingress INGRESS-NAME -n NAMESPACE

Run one of the following commands to edit the Service or Ingress object. In the following examples, replace SERVICE-NAME, INGRESS-NAME, and NAMESPACE with the correct values for your use case.

kubectl edit service SERVICE-NAME -n NAMESPACE  
  
kubectl edit ingress INGRESS-NAME -n NAMESPACE

Review the AWS Load Balancer Controller pod's logs for additional information

To review the AWS Load Balancer Controller logs, run the following command:

kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller

If there's an issue, then you see a Reconciler error. You also see a detailed error message that states why an ingress object or load balancer service fails to create or update. This failure likely occurs because of the following reasons:

• If the error occurs when the controller tries to make AWS API calls, then this relates to a permission or a connectivity issue. Review the controller's AWS Identity and Access Management (IAM) permissions. Then, make sure that security groups or network access control lists (network ACLs) don't explicitly deny outbound connections.
• If the error occurs in the object's configuration, then you have a malformed ingress or service spec or incorrect annotations. Review the annotations for Application Load Balancer or Network Load Balancer on GitHub.

If none of the controller pods show logs, then make sure that the controller pods are running:

kubectl get deployment -n kube-system aws-load-balancer-controller

Verify that a Fargate profile is created for the namespace where the Ingress or Service object is

When target pods are running on Fargate, you must include the IP target type. To verify that you have a Fargate profile for the namespace where the Ingress or Service object is, run the following commands. Replace CLUSTER-NAME with the name of your cluster:

eksctl get fargateprofile --cluster CLUSTER-NAME -o yaml

To create a Fargate profile, run the following command. Replace CLUSTER-NAME, REGION, FARGATE-PROFILE-NAME, and NAMESPACE with the correct values for your use case:

eksctl create fargateprofile --cluster CLUSTER-NAME --region REGION --name FARGATE-PROFILE-NAME --namespace NAMESPACE

Check for unaddressed dependencies

Amazon EKS adds the following rules to the node's security group:

• An inbound rule for client traffic.
• An inbound rule for each load balancer subnet in the VPC for each Network Load Balancer that you create (for health checks).

If Amazon EKS tries to create rules that exceed the maximum number of rules for a security group, then your load balancer deployment might fail.

To make sure that you meet all dependencies, review the relevant documentation. For Application Load Balancer, see Application load balancing on Amazon EKS. For Network Load Balancer, see Network load balancing on Amazon EKS. For example, if you use an Application Load Balancer, then the Service object must specify the NodePort or LoadBalancer to use instance traffic mode.