How do I troubleshoot issues with the API server endpoint of my Amazon EKS cluster?

Last updated: 2020-01-27

I changed the endpoint access setting from public to private on my Amazon Elastic Kubernetes Service (Amazon EKS) cluster. Now, my cluster is stuck in the "failed" state, or I can't run kubectl commands.

Short Description

If you have issues with your Kubernetes API server endpoint, complete the steps in one of the following sections:

  • Your cluster is stuck in the "failed" state and you can't change the endpoint access setting from public to private
  • You can't run kubectl commands on the cluster after changing the endpoint access from public to private

Note: To set up access to the Kubernetes API server endpoint, see How do I set up public and private access to the API server in Amazon EKS?

Resolution

Your cluster is stuck in the "failed" state and you can't change the endpoint access setting from public to private

Your cluster could be in the "failed" state because of a permission issue with AWS Identity and Access Management (IAM).

1.    Confirm that the IAM role for the user is authorized to perform the AssociateVPCWithHostedZone action.

Note: If the action isn't blocked, check if the user's account has AWS Organizations policies that are blocking the API calls and causing the cluster to fail.

2.    Confirm that the IAM user's permission isn't blocked at any level above the account, either implicitly (by not being included in an Allow policy statement) or explicitly (by being included in a Deny policy statement).

Note: Permission is blocked even if the account administrator attaches the AdministratorAccess IAM policy with */* permissions to the IAM user. The permissions for IAM entities are overridden by the permissions from Organizations policies.

You can't run kubectl commands on the cluster after changing the endpoint access from public to private

1.    Confirm that you're using a bastion host or connected networks (such as peered VPCs, AWS Direct Connect, or VPNs) to access the Amazon EKS API endpoint.

Note: In private access mode, you can access the Amazon EKS API endpoint only from within the cluster's VPC.

2.    Check if security groups or network access control lists are blocking the API calls.

If you access your cluster across a peered VPC, confirm that the security groups allow access from the peered VPC to the security group of the control plane at port 443. Also, verify that both the peered VPCs have port 53 open to each other (used for DNS resolution).