My Amazon EMR application fails with an HTTP 403 "Access Denied" AmazonS3Exception

Last updated: 2019-11-21

When I submit an application to an Amazon EMR cluster, the application fails with an HTTP 403 "Access Denied" AmazonS3Exception:

java.io.IOException: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8B28722038047BAA; S3 Extended Request ID: puwS77OKgMrvjd30/EY4CWlC/AuhOOSNsxfI8xQJXMd20c7sCq4ljjVKsX4AwS7iuo92C9m+GWY=), S3 Extended Request ID: puwS77OKgMrvjd30/EY4CWlC/AuhOOSNsxfI8xQJXMd20c7sCq4ljjVKsX4AwS7iuo92C9m+GWY=

Short Description

This error indicates that the application tried to perform an Amazon Simple Storage Service (Amazon S3) operation that failed because of a problem with one of the following:

  • Credentials or role specified in your application code
  • Policy attached to the Amazon Elastic Compute Cloud (Amazon EC2) instance profile role
  • Amazon S3 VPC endpoint policy
  • Amazon S3 source and destination bucket policies

Resolution

Run the following command on the EMR cluster's master node. Replace s3://awsexamplebucket/abc/ with your Amazon S3 path.

aws s3 ls s3://awsexamplebucket/abc/

If this command is successful, it indicates that the credentials or role specified in your application code are causing the "Access Denied" error. The credentials or role must have access to the Amazon S3 path.

If this command fails, check the following to resolve the "Access Denied" error:

Check the policy for the Amazon EC2 instance profile role

By default, applications inherit Amazon S3 access from the AWS Identity and Access Management (IAM) role for the Amazon EC2 instance profile. The IAM policies attached to this role must allow the required Amazon S3 operations on the source and destination buckets.

If you're using EMRFS role mapping, then your application inherits Amazon S3 permissions from the IAM role for the user who submitted the application. This IAM user must have an IAM policy that allows the required Amazon S3 operations on the source and destination buckets.

Check the Amazon S3 VPC endpoint policy

If the EMR cluster's subnet route table has a route to an Amazon S3 VPC endpoint, confirm that the endpoint policy allows the required Amazon S3 operations. You can use the AWS Command Line Interface (AWS CLI) or Amazon VPC console to check and modify the endpoint policy.

AWS CLI:

Run the following command to review the endpoint policy. Replace vpce-9f28e4f6 with your VPC ID.

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids "vpce-9f28e4f6"

If necessary, run the following command to upload a modified endpoint policy. In the example, replace the VPC ID and JSON file path.

aws ec2 modify-vpc-endpoint --vpc-endpoint-id "vpce-9f28e4f6" --policy-document file://policy.json

Amazon VPC console:

1.    Open the Amazon VPC console.

2.    In the navigation pane, choose Endpoints.

3.    Select the Amazon S3 endpoint (the one that's on the EMR cluster's subnet route table), and then choose the Policy tab to review the endpoint policy.

4.    To add the required Amazon S3 actions, choose Edit Policy.

Check the S3 bucket policies

Bucket policies specify which actions are allowed or denied for which principals. The bucket policies for the source and destination buckets must allow the EC2 instance profile role or the mapped IAM role to perform the required Amazon S3 operations. You can use the AWS CLI or Amazon S3 console to check and modify the bucket policies.

AWS CLI:

Run the following command to review a bucket policy. Replace awsexamplebucket with the name of the source or destination bucket.

aws s3api get-bucket-policy --bucket awsexamplebucket

If necessary, run the following command to upload a modified bucket policy. In the example, replace the bucket name and JSON file path.

aws s3api put-bucket-policy --bucket awsexamplebucket --policy file://policy.json

Amazon S3 console:

1.    Open the Amazon S3 console.

2.    Choose the bucket.

3.    Choose the Permissions tab.

4.    Choose Bucket Policy to review and modify the bucket policy.

Accessing S3 buckets in another account

If your application accesses an S3 bucket that belongs to another AWS account, the account owner must allow your IAM role on the bucket policy. For example, the following bucket policy gives all IAM roles and users in emr-account full access to s3://awsexamplebucket/myfolder/.

{
    "Id": "MyCustomPolicy",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowRootAndHomeListingOfCompanyBucket",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::emr-account:root"
                ]
            },
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::awsexamplebucket"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:prefix": [
                        "",
                        "myfolder/"
                    ],
                    "s3:delimiter": [
                        "/"
                    ]
                }
            }
        },
        {
            "Sid": "AllowListingOfUserFolder",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::emr-account:root"
                ]
            },
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::awsexamplebucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "myfolder/*"
                    ]
                }
            }
        },
        {
            "Sid": "AllowAllS3ActionsInUserFolder",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::emr-account:root"
                ]
            },
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::awsexamplebucket/myfolder/*",
                "arn:aws:s3:::awsexamplebucket/myfolder*"
            ]
        }
    ]
}