How can I set up cross-account access for Amazon EMRFS?

Last updated: 2022-08-31

I want to use the Amazon EMR File System (EMRFS) to write to Amazon Simple Storage Service (Amazon S3) buckets that are in a different AWS account.

Short description

Use one of the following options to set up cross-account access for EMRFS:

  • Add a bucket policy. Add a bucket policy for the destination bucket that grants access to the Amazon EMR account. This is easiest option. However, the destination account doesn't own the objects that EMRFS writes to the destination bucket.
  • Use a custom credentials provider. This option allows you to assume an AWS Identity and Access Management (IAM) role in the destination bucket account. This means that the destination account owns objects that EMRFS writes to the destination bucket.
  • Use role mappings in a security configuration. This option also allows EMRFS to assume an IAM role in the destination bucket account. This is the method that's discussed in this article.

Resolution

When you use a security configuration to specify IAM roles for EMRFS, you set up role mappings. A role mapping specifies an IAM role that corresponds to an identifier. An identifier determines the basis for access to Amazon S3 through EMRFS. Identifiers can be users, groups, or Amazon S3 prefixes that indicate a data location. When EMRFS makes a request that matches the basis for access, EMRFS has cluster EC2 instances assume the corresponding IAM role for the request. The IAM permissions attached to that role apply, instead of the IAM permissions attached to the service role for cluster EC2 instances. For more information, see Configure IAM roles for EMRFS requests to Amazon S3.

In the following steps, an identifier is specified as an Amazon S3 prefix that is accessed through EMRFS. This creates cross-account access for EMRFS using a security configuration with role mapping:

1.    Create an IAM role in the destination account. This is the role that you will assume from the EMR cluster.

2.    Add a trust policy similar to the following. The trust policy must allow the Amazon Elastic Compute Cloud (Amazon EC2) role for Amazon EMR to assume the role that you created in step 1. For more information, see Configure roles.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::EMRFSAcctID:role/EMR_EC2_DefaultRole"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

3.    Use the AWS Command Line Interface (AWS CLI) to create a security configuration with a role mapping. The role mapping must specify the role in the destination account (the role that you created in step 1).

Note: You must use the AWS CLI or an SDK to create the security configuration. The console doesn't list roles in other accounts, even if you have permissions to assume those roles. If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

Supply a JSON object similar to the following for the role mapping. Replace these values in the example:

arn:aws:iam::DestinationAcctID:role/role_in_destination_account: the Amazon Resource Name (ARN) of the role that you created in step 1
s3://doc-example-bucket/: the bucket that you want EMRFS to write to

{
  "AuthorizationConfiguration": {
    "EmrFsConfiguration": {
      "RoleMappings": [
        {
          "Role": "arn:aws:iam::DestinationAcctID:role/role_in_destination_account",
          "IdentifierType": "Prefix",
          "Identifiers": [
            "s3://doc-example-bucket/"
          ]
        }
      ]
    }
  }
}

4.    Create an IAM policy and then attach it to the Amazon EMR EC2 instance profile (for example, EMR_EC2_DefaultRole).

The following example policy allows AWS Security Token Service (STS) to assume all roles. At a minimum, your policy must allow STS to assume the role that you created in step 1. For more information, see Granting permissions to create temporary security credentials.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "*"
    }
  ]
}

5.    Launch an EMR cluster and specify the security configuration that you created in step 3.

Note: If the destination bucket uses server-side encryption with AWS Key Management Service (AWS KMS), then the assumed role must be a key user in the AWS KMS customer managed key. You can't access the bucket if the role isn't listed in the AWS KMS key.


Did this article help?


Do you need billing or technical support?