How do I resolve kubelet or CNI plugin issues for Amazon EKS?

Last updated: 2021-11-15

I want to resolve issues with my kubelet or CNI plugin for Amazon Elastic Kubernetes Service (Amazon EKS).

Short description

To assign and run an IP address to the pod on your worker node with your CNI plugin (on the Kubernetes website), you must have the following:

  • AWS Identity and Access Management (IAM) permissions, including a CNI policy attached to your worker node's IAM role or provided through service account IAM roles
  • An Amazon EKS API server endpoint that can be reached from the worker node
  • Network access to API endpoints for Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Registry (Amazon ECR), and Amazon Simple Storage Service (Amazon S3)
  • Enough available IP addresses in your subnet
  • A kube-proxy that's running successfully for the aws-node pod to progress into Ready status

Resolution

Verify that the aws-node pod is in Running status on each worker node

To verify that the aws-node pod is in Running status on a worker node, run the following command:

kubectl get pods -n kube-system -l k8s-app=aws-node -o wide

If the command output shows that the RESTARTS count is 0, then the aws-node pod is in Running status. Try the troubleshooting steps in the Verify that your subnet has enough free IP addresses available section.

If the command output shows that the RESTARTS count is greater than 0, then try the following steps:

Verify that the worker node can reach the API server endpoint of your Amazon EKS cluster:

curl -vk https://eks-api-server-endpoint-url

Verify connectivity to your Amazon EKS cluster

1.    Verify that your worker node's security group settings for Amazon EKS are correctly configured. For more information, see Amazon EKS security group considerations.

2.    Verify that your worker node's network access control list (ACL) rules for your subnet allow communication with the Amazon EKS API server endpoint.

Important: Allow inbound and outbound traffic on port 443.

3.    Verify that the kube-proxy pod is in Running status on each worker node:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

4.    Verify that your worker node can access API endpoints for Amazon EC2, Amazon ECR, and Amazon S3.

Note: You can configure these services through public endpoints or AWS PrivateLink.

Verify that your subnet has enough free IP addresses available

To list available IP addresses in each subnet in the Amazon Virtual Private Cloud (Amazon VPC) ID, run the following command:

aws ec2 describe-subnets --filters "Name=vpc-id,Values= VPCID" | jq '.Subnets[] | .SubnetId + "=" + "\(.AvailableIpAddressCount)"'

Note: The AvailableIpAddressCount should be greater than 0 for the subnet where the pods are launched.

Check whether your security group limits have been reached

Your pod networking configuration can fail if you reach the limits of your security groups per elastic network interface.

For more information, see Amazon VPC quotas.

Verify that you're running the latest stable version of the CNI plugin

To confirm that you have the latest version of the CNI plugin, see Managing the Amazon VPC CNI add-on.

For additional troubleshooting, see the AWS GitHub issues page and release notes for the CNI plugin.

Check the logs of the VPC CNI plugin on the worker node

If you created a pod and an IP address didn't get assigned to the container, then you receive the following error:

failed to assign an IP address to container

To check the logs, go to the /var/log/aws-routed-eni/ directory, and then locate the file names plugin.log and ipamd.log.

Verify that your kubelet pulls the docker container images

If your kubelet doesn't pull the docker container images for the kube-proxy and amazon-k8s-cni containers, then you receive the following error:

network plugin is not ready: cni config uninitialized

Make sure that the EKS API server endpoint can be reached from the worker node.

Verify that the WARM_PREFIX_TARGET value is set correctly

WARM_PREFIX_TARGET must be set to a value greater than or equal to 1. If it's set to 0, then you receive the following error:

Error: Setting WARM_PREFIX_TARGET = 0 is not supported while WARM_IP_TARGET/MINIMUM_IP_TARGET is not set. 
Please configure either one of the WARM_{PREFIX/IP}_TARGET or MINIMUM_IP_TARGET env variable

See CNI plugin Configuration variables for more information.

Check the reserved space in the subnet

Make sure that you have enough available /28 IP CIDR (16 IPs) blocks in the subnet. All 16 IPs must be contiguous. If you don't have a /28 range of continuous IPs, then you receive the following error:

InsufficientCidrBlocks

To resolve the error, create a new subnet and launch the pods from there. You can also use an Amazon EC2 subnet CIDR reservation to reserve space within a subnet with an assigned prefix. For more information, see Subnet CIDR reservations.


Did this article help?


Do you need billing or technical support?