Why do I get the "webidentityerr" error when using AWS Load Balancer Controller in Amazon EKS?

Last updated: 2021-12-08

I receive the following error message when I try to use the AWS Load Balancer Controller in Amazon Elastic Kubernetes Service (Amazon EKS):

"failed to find existing LoadBalancer due to WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403"

How do I resolve this error?

Short description

When using the AWS Load Balancer Controller in Amazon EKS, the WebIdentityErr:  failed to retrieve credentials caused by: AccessDenied error can occur for the following reasons:

  • Incorrect service account configurations
  • Incorrect trust relationship of the AWS Identity and Access Management (IAM) role used in the service account

With AWS Load Balancer Controller, worker nodes perform the tasks. These worker nodes must be granted access to the AWS Application Load Balancer or AWS Network Load Balancer resources using IAM permissions. The IAM permissions can either be set up using IAM roles for the service account, or they can be attached directly to the worker node's IAM roles. For more information, see Load Balancer Controller installation on the Kubernetes website.

Resolution

Incorrect service account configurations

To check whether your service account configurations are properly configured, perform the following steps:

1.    Verify the service account (SA) name defined in your deployment:

kubectl describe deploy <aws-load-balancer-controller> -n kube-system | grep -i "Service Account"

2.    Describe the service account:

kubectl describe sa <aws-load-balancer-controller> -n kube-system

3.    Verify the SA annotation for the IAM role:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/<AmazonEKSLoadBalancerControllerRole>

4.    If this annotation is missing or incorrect, then update the annotation. Make sure that you properly associated the IAM role to a service account, like this:

kubectl annotate serviceaccount -n <SERVICE_ACCOUNT_NAMESPACE> <SERVICE_ACCOUNT_NAME> \ eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>

5.    Delete and recreate any existing Amazon EKS pods that are associated with the service account. Be sure to apply the credential environment variables:

kubectl delete pods <aws-loadbalancer-controller-pod> -n <kube-system>

Incorrect trust relationship between the AWS Identity Access Management (IAM) role used and service account

The following examples show some common mistakes that can occur when establishing the trust relationship between your IAM role and service account.

Example 1: IAM role or trust relationship isn't properly defined for the "sts:AssumeRoleWithWebIdentity" action

Check to make sure that the trust relationship is properly defined for the "sts:AssumeRoleWithWebIdentity" action (and not the "sts:AssumeRole" action). 

Here's an example of a trust relationship that isn't properly defined:

{
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "xxxxx.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
}

To correct the trust relationship, make sure to define the trust relationship for the "sts:AssumeRoleWithWebIdentity" action:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS-ACCOUNT>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>": "system:serviceaccount:kube-system:<aws-load-balancer-controller-serviceaccount>"
        }
      }
    }
  ]
}

Note: Make sure to replace all variables with your own values.

Example 2: Incorrect OIDC provider ID when creating an Amazon EKS cluster

Make sure to properly create and verify an OIDC provider for your Amazon EKS cluster. Verify that the OIDC provider ID and the associated AWS Region are correctly listed. Otherwise, you receive a WebIdentityErr error.

Example 3: Incorrect service account (SA) name and its namespace when configuring the AWS Load Balancer Controller deployment

Make sure to enter the correct SA name and its namespace when you update your AWS Load Balancer Controller deployment.

Example 4: Missing "sts.amazonaws.com" steps from the trust relationship

If the service role associated with your EKS pod is unable to perform the STS operation on the "AssumeRoleWithWebIdentity" action, then update the trust relationship. The trust relationship must include "sts.amazonaws.com" to perform an STS operation.

For example:

"Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>:sub": "system:serviceaccount:kube-system:<aws-load-balancer-controller-serviceaccount>",
          "oidc.eks.<REGION>.amazonaws.com/id/<Your-EKS-cluster-OIDC-Provider-ID>:aud": "sts.amazonaws.com"
        }
}

For more information about IAM conditions with multiple keys or values, see Creating a condition with multiple keys or values.


Did this article help?


Do you need billing or technical support?