How can I get my worker nodes to join my Amazon EKS cluster?

Last updated: 2022-07-29

My worker nodes won't join my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Short description

To get your worker nodes to join your Amazon EKS cluster, you must complete the following:

  • Confirm that you have DNS support for your Amazon Virtual Private Cloud (Amazon VPC).
  • Get the right permissions for your instance profile's worker nodes.
  • Configure the user data for your worker nodes.
  • Verify that your worker nodes are in a subnet that's associated with your EKS cluster.
  • Update the aws-auth ConfigMap with the NodeInstanceRole of your worker nodes.
  • Meet the security group requirements of your worker nodes.
  • Set the tags for your worker nodes.
  • Confirm that your worker nodes can reach the API server endpoint for your EKS cluster.
  • For AWS Regions that support AWS Security Token Service (AWS STS) endpoints, confirm that the Regional AWS STS endpoint is activated.
  • Connect to your worker node's Amazon Elastic Compute Cloud (Amazon EC2) instance using SSH, and search through kubelet agent logs for errors.

Important: The following steps don't include the configurations that are required to register worker nodes in your cluster in environments where the following criteria aren't met:

  • In the VPC for your cluster, the configuration parameter domain-name-servers is set to AmazonProvidedDNS. For more information, see DHCP options sets.
  • You're using an Amazon EKS-optimized Linux Amazon Machine Image (AMI) to launch your worker nodes.
    Note: The Amazon EKS-optimized Linux AMI provides all necessary configurations, including a /etc/eks/bootstrap.sh bootstrap script for registering worker nodes to your cluster.

Resolution

Identify common issues using the AWS Systems Manager automation runbook

Use the AWSSupport-TroubleshootEKSWorkerNode runbook to find common issues that prevent worker nodes from joining your cluster.

Important: For the automation to work, your worker nodes must have permission to access Systems Manager and have Systems Manager running. To grant permission, attach the AmazonSSMManagedInstanceCore AWS managed policy to the AWS Identity and Access Management (IAM) role that corresponds to your EC2 instance profile. This is the default configuration for EKS managed node groups that are created through eksctl.

  1. Open the runbook.
  2. Check that the AWS Region in the AWS Management Console is set to the same Region as your cluster.
    Note: Review the Document details section of the runbook for more information about the runbook.
  3. In the Input parameters section, specify the name of your cluster in the ClusterName field and EC2 instance ID in the WorkerID field.
  4. (Optional) In the AutomationAssumeRole field, specify the IAM role to allow Systems Manager to perform actions. If it's not specified, then the IAM permissions of your current IAM entity is used to perform the actions in the runbook.
  5. Choose Execute.
  6. Check the Outputs section to see why your worker node isn't joining your cluster and steps that you can take to resolve it.

Confirm that you have DNS support for your Amazon VPC

Confirm that the Amazon VPC for your EKS cluster has support for a DNS hostname and DNS resolution.

If required, view and update the DNS support attributes for your VPC.

Get the right permissions for your instance profile's worker nodes

Attach the following AWS managed polices to the role that's associated with your instance profile's worker nodes:

  • AmazonEKSWorkerNodePolicy
  • AmazonEKS_CNI_Policy
  • AmazonEC2ContainerRegistryReadOnly

To attach policies to roles, see Adding IAM identity permissions (console).

Configure the user data for your worker nodes

Note: If you're using AWS CloudFormation to launch your worker nodes, then you don't have to configure the user data for your worker nodes. Instead, follow the instructions for launching self-managed Amazon Linux nodes using the AWS Management Console.

If you launch your worker nodes using managed node groups, then you're not required to configure any user data with Amazon EKS optimized Amazon Linux AMIs. You must configure the user data only if you're using custom AMIs to launch your worker nodes through managed node groups.

To configure user data for your worker nodes, specify the user data when you launch your Amazon EC2 instances.

For example, if you’re using a third-party tool such as Terraform, update the User data field to launch your EKS worker nodes with the following:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}

Important: Replace ${ClusterName} with the name of your EKS cluster. Replace ${BootstrapArguments} with additional bootstrap values, or leave this property blank.

Verify that your worker nodes are in same Amazon VPC as your EKS cluster

  1. Open the Amazon EKS console.
  2. Choose Clusters, and then select your cluster.
  3. In the Networking section, identify the subnets that are associated with your cluster.

Note: You can configure different subnets to launch your worker nodes in. The subnets must exist in the same Amazon VPC and be appropriately tagged. Amazon EKS automatically manages tags only for subnets that you configure during cluster creation. Therefore, make sure that you tag the subnets appropriately.

Update the aws-auth ConfigMap with the NodeInstanceRole of your worker nodes

Verify that the aws-auth ConfigMap is correctly configured with your worker node's IAM role, not the instance profile.

Meet the security group requirements of your worker nodes

Confirm that your control plane's security group and worker node security group are configured with the recommended settings for inbound and outbound traffic. Also, confirm that your custom network ACL rules are configured to allow traffic to and from "0.0.0.0/0" for ports 80,443 and 1025-65535.

Set the tags for your worker nodes

For the Tag property of your worker nodes, set Key to kubernetes.io/cluster/clusterName and set Value to owned.

For more information, see Cluster VPC considerations.

Confirm that your worker nodes can reach the API server endpoint for your EKS cluster

Consider the following:

  • You can launch worker nodes in a subnet that's associated with a route table that routes to the API endpoint through a NAT or internet gateway.
  • If you launch your worker nodes in a restricted private network, then confirm that your worker nodes can reach the EKS API server endpoint.
  • If you launch worker nodes with an Amazon VPC that uses a custom DNS instead of AmazonProvidedDNS, then they might not resolve the endpoint. An unresolved endpoint happens when public access to the endpoint is deactivated, and only private access is activated. For more information, see Enabling DNS resolution for Amazon EKS cluster endpoints.
  • For private clusters, you must include the VPC endpoints that are specific to private clusters. Make sure that self-managed and managed nodes have access to the Amazon VPC endpoints. For managed node groups, the Amazon VPC endpoint security group must allow the CIDR for the subnets or the cluster security group.

Confirm that Regional STS endpoints are activated

If the cluster is in a Region that supports STS endpoints, then activate the Regional STS endpoint to authenticate the kubelet. The kubelet can then create the node object.

Connect to your EKS worker node instance with SSH and check kubelet agent logs

The kubelet agent is configured as a systemd service.

1.FSPTo validate your kubelet logs, run the following command:

journalctl -f -u kubelet

2.FSPTo resolve any issues, check the Amazon EKS troubleshooting guide for common errors.


Did this article help?


Do you need billing or technical support?