How can I troubleshoot issues when creating a load balancer using the AWS Load Balancer Controller?

Last updated: 2023-01-27

I can't create a Network Load Balancer or an Application Load Balancer using the AWS Load Balancer Controller. Or, the load balancer isn't created after I created the Service or Ingress objects.

Short description

To troubleshoot load balancer creation issues, do the following:

  • Make sure that all prerequisites are met.
  • Check the annotations of the Ingress (Application Load Balancer) or Service (Network Load Balancer) object.
  • Review the AWS Load Balancer Controller pod's logs for additional information.
  • If the cluster runs on AWS Fargate, then verify that a Fargate profile is created for the namespace where the Ingress or Service object resides.
  • Check for unaddressed dependencies.

The AWS Load Balancer Controller manages Elastic Load Balancing for an EKS cluster. The controller provisions the following resources:

  • An Application Load Balancer when you create a Kubernetes ingress.
  • A Network Load Balancer when you create a Kubernetes service of type LoadBalancer. Previously, the Kubernetes Network Load Balancer was used for instance targets, and the AWS Load Balancer Controller was used for IP targets. With the AWS Load Balancer Controller version 2.3.0 or later, you can create a Network Load Balancer using either target type. For more information, see Target type.

Resolution

Make sure that all prerequisites are met

For a list of Application Load Balancer prerequisites, see Application load balancing on Amazon EKS. For a list of Network Load Balancer prerequisites, see Network load balancing on Amazon EKS.

1.    Verify that the AWS Load Balancer Controller is successfully provisioned. It's a best practice to use version 2.4.4 or later.

2.    Check the number of subnets. Application Load Balancer needs at least two subnets in different Availability Zones. Network Load Balancer needs at least one subnet. The subnets must have at least eight available IP addresses. For more information, see View your VPCs.

3.    You must use the following tag in certain scenarios:

  • Key: "kubernetes.io/cluster/cluster-name"
  • Value: "shared" or "owned"

If you're using an Application Load Balancer

You must tag exactly one security group in the following scenarios:

  • You're using multiple security groups attached to worker node.
  • You're using the AWS Load Balancer Controller version v2.1.1 or earlier.

If you're using a Network Load Balancer

If you're using the AWS Load Balancer Controller version v2.1.1 or earlier, then subnets must be tagged.

For information on adding tags from the Amazon EC2 console, see Work with tags using the console. For information on adding tags using the AWS Command Line Interface (AWS CLI), see Work with tags using the command line.

Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

4.FSPUnless subnet IDs are explicitly specified as annotations in the Service object or Ingress object, make sure that the subnets have the following tags. Without these tags, Subnet Auto Discovery doesn't work.

Private subnets tag:

  • Key: "kubernetes.io/role/internal-elb"
  • Value: "1"

Public subnets tag:

  • Key: "kubernetes.io/role/elb"
  • Value: "1"

Check the annotations of the Ingress or Service object

Verify annotations on the Service object or annotations on the Ingress object. The annotations needed to configure a load balancer are as follows:

Note: Other annotations use default values. For a list of all available annotations supported by the AWS Load Balancer Controller for Application Load Balancing, see Ingress annotations on GitHub. For a list of all available annotations supported by the AWS Load Balancer Controller for Network Load Balancing, see Service annotations on GitHub.

Application Load Balancer

Network Load Balancer

  • With IP targets: service.beta.kubernetes.io/aws-load-balancer-type: "external" service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
  • With instance targets: service.beta.kubernetes.io/aws-load-balancer-type: "external" service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"

Run one of the following commands to view the Service or the Ingress object. In the following example, replace SERVICE-NAME/INGRESS-NAME and NAMESPACE with the correct values for your use case.

kubectl describe service <SERVICE-NAME> -n <NAMESPACE> 

kubectl describe ingress <INGRESS-NAME> -n <NAMESPACE>

Run one of the following commands to edit the Service or Ingress object. In the following examples, replace SERVICE-NAME/INGRESS-NAME and NAMESPACE with the correct values for your use case.

kubectl edit service <SERVICE-NAME> -n <NAMESPACE>

kubectl edit ingress <INGRESS-NAME> -n <NAMESPACE>

Review the AWS Load Balancer Controller pod's logs for additional information

Run the following command to review the AWS Load Balancer Controller logs:

kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller

If none of the controller pods show logs, then make sure that the controller pods are running:

kubectl get deployment -n kube-system aws-load-balancer-controller

If the cluster runs on Fargate, then verify that a Fargate profile is created for the namespace where the Ingress or Service object resides

The IP target type is required when target pods are running on Fargate. Run the following commands to verify that a Fargate profile is created for the namespace where the Ingress or Service object resides. In the following example, replace CLUSTER-NAME with the name of your cluster.

eksctl get fargateprofile --cluster <CLUSTER-NAME> -o yaml

To create a Fargate profile, run the following command. In the following example, replace CLUSTER-NAME, REGION, FARGATE-PROFILE-NAME, and NAMESPACE with the correct values for your use case.

eksctl create fargateprofile --cluster <CLUSTER-NAME> --region <REGION> --name <FARGATE-PROFILE-NAME> --namespace <NAMESPACE>

Check for unaddressed dependencies

Amazon EKS adds the following rules to the node's security group:

  • An inbound rule for client traffic
  • An inbound rule for each load balancer subnet in the VPC for each Network Load Balancer that you create (for health checks).

Deploying a service of type LoadBalancer might fail if Amazon EKS attempts to create rules that exceed the quota for the maximum number of rules allowed for a security group.

Review the documentation to make sure that all dependencies are met. For Application Load Balancer, see Application load balancing on Amazon EKS. For Network Load Balancer, see Network load balancing on Amazon EKS. For example, If you're using an Application Load Balancer, then the Service object must specify the NodePort or LoadBalancer to use instance traffic mode.


Did this article help?


Do you need billing or technical support?