How can I troubleshoot IAM permission access denied or unauthorized errors?

Last updated: 2019-12-20

I am trying to access an AWS resource and I received an "access denied" or "unauthorized" error. How can I troubleshoot permission issues for an AWS Identity and Access Management (IAM) user?

Short Description

You can use Amazon Athena queries or the AWS Command Line Interface (AWS CLI) to troubleshoot IAM permission API call failures.

Resolution

Option 1: Use Athena queries to troubleshoot IAM permission API call failures by searching AWS CloudTrail logs

Note: Before you begin, you must have a trail enabled to log to an Amazon Simple Storage Service (Amazon S3) bucket. This is because Athena uses events recorded in CloudTrail log files that are delivered to an Amazon S3 bucket for that trail.

1.    Follow the steps in the create the Athena table section of How do I automatically create tables in Amazon Athena to search through AWS CloudTrail logs?

Note: Athena tables that are created automatically are in the same AWS Region as your Amazon S3 bucket.

2.    Open the Athena console, choose New query, and then choose the dialog box to clear the sample query.

3.    Enter the following example query, and then choose Run Query.

In this example query, the time format uses ISO 8601 basic format with the Z variable for UTC.

Note: Replace your-arn with the IAM Amazon Resource Names (ARN) for your resources and your-table with your table name.

SELECT from_iso8601_timestamp(eventTime) AS "Time", useridentity.arn AS "Identity ARN", eventID AS "Event ID",
         eventsource AS "Service", eventname AS "Action", errorCode AS "Error", errorMessage AS "Message"
FROM your-table
WHERE from_iso8601_timestamp(eventtime) >= from_iso8601_timestamp('2019-10-29T06:40:00Z')
        AND from_iso8601_timestamp(eventtime) < from_iso8601_timestamp('2019-10-29T06:55:00Z')
        AND userIdentity.arn = 'your-arn'
        AND eventType = 'AwsApiCall'
        AND errorCode is not null
        AND (lower(errorCode) LIKE '%accessdenied%' OR lower(errorCode) LIKE '%unauthorized%')
ORDER BY eventTime desc

4.    This example table output lists permissions errors for the identity ARN:

| Time                        | Event ID                             | Service                  | Action       | Error        | Message                                                                                                              |
|-----------------------------|--------------------------------------|--------------------------|--------------|--------------|----------------------------------------------------------------------------------------------------------------------|
| 2019-10-29 06:52:45.000 UTC | 0406f0c1-47a8-4f71-8a94-18267b84042a | cloudtrail.amazonaws.com | LookupEvents | AccessDenied | User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents with an explicit deny |
| 2019-10-29 06:41:48.000 UTC | 14e5e77c-f682-45e1-8c88-12d15af293dd | cloudtrail.amazonaws.com | LookupEvents | AccessDenied | User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents                       |

Note: CloudTrail event outputs can take up to 15 minutes to deliver results.

5.    Optionally, get errors for all users by removing this line from the example query:

AND userIdentity.arn = 'your-arn'

6.    Optionally, get all errors from a selected time period by removing this line from the example query:  

AND (lower(errorCode) LIKE '%accessdenied%' OR lower(errorCode) LIKE '%unauthorized%')

Option 2: Use the AWS CLI to troubleshoot IAM permission API call failures

Important: Before you begin, be sure that you have installed and configured the AWS CLI.

Note: This AWS CLI script requires the jq command line JSON processor. For the tutorial and download instructions, see JSON Output Format. For distributions that use the yum package, run the following command:

$ sudo yum install jq

1.    Run the following AWS CLI command:

Note: Replace your-arn with the IAM ARNs for your resources.

( echo "Time,Identity ARN,Event ID,Service,Action,Error,Message";
  aws cloudtrail lookup-events --start-time "2019-10-29T06:40:00Z" --end-time "2019-10-29T06:55:00Z" --query "Events[*].CloudTrailEvent" --output text \
    | jq -r ". | select(.userIdentity.arn == \"your-arn\" and .eventType == \"AwsApiCall\" and .errorCode != null
    and (.errorCode | ascii_downcase | (contains(\"accessdenied\") or contains(\"unauthorized\"))))
    | [.eventTime, .userIdentity.arn, .eventID, .eventSource, .eventName, .errorCode, .errorMessage] | @csv"
) | column -t -s'",'

Note: The rate of lookup requests to CloudTrail is limited to one request per second per account. If you exceed this limit, then a throttling error occurs.

2.    This example table output lists permissions errors for the identity ARN from the past 90 days:

Time                  Event ID                              Service                   Action        Error         Message
2019-10-29T06:52:45Z  0406f0c1-47a8-4f71-8a94-18267b84042a  cloudtrail.amazonaws.com  LookupEvents  AccessDenied  User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents with an explicit deny
2019-10-29T06:41:48Z  14e5e77c-f682-45e1-8c88-12d15af293dd  cloudtrail.amazonaws.com  LookupEvents  AccessDenied  User: arn:aws:iam::account:user/username is not authorized to perform: cloudtrail:LookupEvents

3.    Optionally, get errors for all users by removing this line:

.userIdentity.arn == \"your-arn\" and 

4.    Optionally, get all errors from the selected time period by removing this line:

and (.errorCode | ascii_downcase | (contains(\"accessdenied\") or contains(\"unauthorized\")))

Troubleshoot unauthorized errors

Athena and the AWS CLI example outputs regard calls to the LookupEvents action in the CloudTrail service.

The cloudtrail:LookupEvents with an explicit deny output indicates that the IAM policy is incorrect. All IAM policy types can return an explicit deny error. And explicit deny statements always overrides allow statements.

The cloudtrail:LookupEvents output indicates that the error is a result of an implicit deny. This output error can occur if:

  • The IAM policy explicitly allows the cloudtrail:LookupEvents action.
  • The maximum permissions for an identity is limited by an IAM policy.
  • The AWS Organizations service control policy (SCP) for the account lacks an explicit allow.

Policy types evaluated by IAM to establish access are:

  • AWS Organizations SCPs
  • Resource-based policies
  • IAM permissions boundaries
  • Session policies
  • Identity-based policies

For additional information about how IAM policies are evaluated and managed, see Policy Evaluation Logic and Managing IAM Policies.