How do I troubleshoot service load balancers for Amazon EKS?

Last updated: 2020-01-24

I can't create a Kubernetes service that's backed by a load balancer in Amazon Elastic Kubernetes (Amazon EKS).

Short Description

To troubleshoot your service load balancer, verify that you have the following:

  • The correct tags for your Amazon Virtual Private Cloud (Amazon VPC) subnets
  • The required AWS Identity and Access Management (IAM) permissions for your cluster's IAM role
  • A valid Kubernetes service definition
  • Load balancers that stay within your account limit
  • Enough free IP addresses on your subnets

If you're still having an issue after verifying all the preceding items, then follow the steps in the Troubleshooting section.

Resolution

Note: The following steps apply to the Classic Load Balancer and the Network Load Balancer. For the Application Load Balancer, see ALB Ingress Controller on Amazon EKS.

The correct tags for your Amazon VPC subnets

1.    Open the AWS VPC console.

2.    On the navigation pane, choose Subnets.

3.    Choose the Tags tab for each subnet, and then confirm that a tag exists. See the following example:

Key: kubernetes.io/cluster/yourEKSClusterName
Value: shared

4.    For your public subnets, confirm that the following tag exists:

Key: kubernetes.io/role/elb
Value: 1

Note: To see if a subnet is a public subnet, check the route table associated with the subnet. A public subnet has a route to an internet gateway (igw-xxxxxxxxxxx). A private subnet has a route to the internet through a NAT gateway or NAT instance, or no route to the internet at all.

Important: You must have the tag in step 4 to create an internet-facing load balancer service.

5.    For your private subnets, confirm that the following tag exists:

Key: kubernetes.io/role/internal-elb
Value: 1

Important: You must have the tag in step 5 to create an internal-facing load balancer service.

The required IAM permissions for your cluster's IAM role

1.    Open the Amazon EKS console.

2.    On the navigation pane, choose Clusters.

3.    Select your cluster, and then note your Cluster IAM Role ARN.

4.    Open the IAM console.

5.    On the navigation pane, choose Roles.

6.    Select the role that matches the Cluster IAM Role ARN that you identified in step 3.

7.    Confirm that the AWS managed policy AmazonEKSClusterPolicy is attached to your role.

Note: The Amazon EKS control plane assumes the preceding IAM role to create a load balancer for your service.

A valid Kubernetes service definition

1.    In the YAML file for your Kubernetes service, verify that spec.type is set to LoadBalancer.

Here's an example of a Kubernetes service backed by a load balancer:

apiVersion: v1
kind: Service
metadata:
  annotations:
    # This annotation is only required if you are creating an internal facing ELB. Remove this annotation to create public facing ELB.
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  name: nginx-elb
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - name: "http"
    port: 80
    targetPort: 80
  selector:
    app: nginx

Note: To customize your service with a different annotation, see Internal load balancer and TLS support on AWS.

Load balancers that stay within your account limit

An AWS account has a maximum of 20 load balancers per AWS Region by default.

To check how many load balancers that you have, open the Amazon EC2 console, and then choose Load Balancers from the navigation pane.

If you have reached the maximum number of load balancers, then you can apply for an increase with Service Quotas.

Enough free IP addresses on your subnets

To create a load balancer, each subnet of that load balancer must have a minimum of eight free IP addresses. This is required for both the Classic Load Balancer and Network Load Balancer.

Troubleshooting

To check the Kubernetes service for an error message that can help you troubleshoot the issue, run the following command:

$ kubectl describe service my-elb-service

If the service is created successfully, the output is similar to the following:

...
...
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  47s   service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   44s   service-controller  Ensured load balancer

If the service isn't created successfully, then you receive an error message.

To get more information about error message, you can:


Did this article help you?

Anything we could improve?


Need more help?