How do I troubleshoot IRSA errors in Amazon EKS?

Last updated: 2021-12-10

I often get errors when I use IAM roles for service accounts (IRSA) with my Amazon Elastic Kubernetes Service (Amazon EKS).

Short description

To troubleshoot issues with IRSA in Amazon EKS, complete the steps in one of the following sections:

  • Check the formatting of the IAM ARN
  • Check if you have an IAM OIDC provider for your AWS account
  • Verify the audience of the OIDC provider
  • Verify that you created the OIDC resource with a root certificate thumbprint
  • Check the configuration of your IAM role's trust policy
  • Verify that your pod identity webhook configuration exists and is valid
  • Verify that you're using supported AWS SDKs

Resolution

Check the formatting of the IAM ARN

If your IAM Amazon Resource Name (ARN) is set in the relative service account annotation and isn't formatted correctly, then you receive the following error:

An error occurred (ValidationError) when calling the AssumeRoleWithWebIdentity
operation: Request ARN is invalid

Example of an incorrect ARN format:

eks.amazonaws.com/role-arn: arn:aws:iam::::1234567890:role/example

The preceding ARN format is incorrect because it has an extra colon ( : ). To make sure that you use the correct ARN format, see IAM ARNs.

Check if you have an IAM OIDC provider for your AWS account

If you haven't created an OpenID Connect (OIDC) provider, then you receive the following error:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://oidc.eks.region.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Get the IAM OIDC provider URL:

aws eks describe-cluster --name cluster name --query "cluster.identity.oidc.issuer" --output text

Note: Replace cluster name with your cluster name.

Example output:

https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E

List the IAM OIDC providers:

aws iam list-open-id-connect-providers | grep EXAMPLED539D4633E53DE1B716D3041E

Note: Replace EXAMPLED539D4633E53DE1B716D3041E with the value that the previous command returned.

If the OIDC provider doesn't exist, then use the following eksctl command to create one:

eksctl utils associate-iam-oidc-provider --cluster cluster name --approve

Note: Replace cluster name with your cluster name.

You can also use the AWS Management Console to create an IAM OIDC provider for your cluster.

Verify the audience of the IAM OIDC provider

When you create an IAM OIDC provider, you must use sts.amazonaws.com as your audience. If the audience is incorrect, then you receive the following error:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Incorrect token audience

Check the audience of the IAM OIDC provider:

aws iam get-open-id-connect-provider --open-id-connect-provider-arn ARN-of-OIDC-provider

Note: Replace ARN-of-OIDC-provider with the ARN of your OIDC provider.

-or-

Complete the following steps:

1.    Open the Amazon EKS console.

2.    Select the name of your cluster and then choose the Configuration tab.

3.    In the Details section, note the value of the OpenID Connect provider URL.

4.    Open the IAM console.

5.    In the navigation pane, under Access Management, choose Identity Providers.

6.    Select the provider that matches the URL for your cluster.

To change the audience, complete the following steps:

1.    Open the IAM console.

2.    In the navigation pane, under Access Management, choose Identity Providers.

3.    Select the provider that matches the URL for your cluster.

4.    Choose Actions and then choose Add audience.

5.    Add sts.amazonaws.com.

Verify that you created the IAM OIDC resource with a root certificate thumbprint

If you didn't use a root certificate thumbprint to create the OIDC provider, then you receive the following error:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

Note: Non-root certificate thumbprints renew yearly, and root certificate thumbprints renew every decade. Therefore, it's a best practice to use a root certificate thumbprint when you create an IAM OIDC.

If you used AWS Command Line Interface (AWS CLI), the AWS Tools for PowerShell, or the IAM API to create your IAM OIDC, then you must manually obtain the thumbprint. If you created your IAM OIDC in the IAM console, then it's still a best practice to manually obtain the thumbprint to verify that the console fetched the correct one.

Obtain the root certificate thumbprint and its expiration date:

echo | openssl s_client -servername oidc.eks.your-region-code.amazonaws.com -showcerts -connect oidc.eks.your-region-code.amazonaws.com:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout -dates | sed 's/://g' | awk -F= '{print tolower($2)}'

Note: Replace your-region-code with the AWS Region that your cluster is located in.

Example output:

9e99a48a9960b14926bb7f3b02e22da2b0ab7280 sep 2 000000 2009 gmt jun 28 173916 2034 gmt

In the preceding example output, 9e99a48a9960b14926bb7f3b02e22da2b0ab7280 is the thumbprint, sep 2 000000 2009 gmt is the certificate start date, and jun 28 173916 2034 is the certificate expiration date.

Check the configuration of your IAM role's trust policy

If the trust policy of the IAM role is misconfigured, then you receive the following error:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

To resolve this issue, make sure that you're using the correct IAM OIDC provider. If the IAM OIDC provider is correct, then use the IAM role configuration guide to check if the trust policy's conditions are correctly configured.

Verify that your pod identity webhook configuration exists and is valid

If you accidentally deleted or changed your webhook configuration, then IRSA stops working.

Run the following command to verify that your webhook configuration exists and is valid:

kubectl get mutatingwebhookconfiguration pod-identity-webhook  -o yaml

Verify that you're using supported AWS SDKs

Make sure that you're using an AWS SDK version that supports assuming an IAM role through the OIDC web identity token file.


Did this article help?


Do you need billing or technical support?