How can I resolve 400 errors with access denied for AWS KMS ciphertext in AWS Glue?

Short description

When using AWS Glue, you might receive this error when you try to access AWS resources like Amazon Simple Service Solution (Amazon S3) buckets:

"The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access."

You receive this error when you access any AWS service that's encrypted with AWS Key Management Service (AWS KMS) keys using AWS Glue jobs or crawlers. The error is most likely to happen for one of these reasons:

• The principal trying to access the encrypted key doesn't have the required AWS KMS permissions
• The AWS KMS key doesn't exist or doesn't exist in a specific AWS Region

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent AWS CLI version.

Check if your AWS Glue role policy has the required permissions

First, check that the AWS Glue role policy you're using has the permissions needed to access the resource's AWS KMS key.

For example, assume that you have an AWS Glue role called AWSGlue-MyGlueCustomRole. This role is accessing an S3 bucket that's encrypted with the AWS KMS key arn:aws:kms:us-east-1:XXXXXXXX9645:key/64d3e826-1a87-4cbc-95b8-fb386730a07. The AWS Glue role policy must have the following context in addition to its other permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:GenerateDataKey"
            ],
            "Resource": "arn:aws:kms:us-east-1:XXXXXXXX9645:key/64d3e826-1a87-4cbc-95b8-fb386730a07"
        }
    ]
}

Check if your AWS KMS key policy allows the AWS Glue role

The AWS KMS key policy must allow the AWS Glue role as part of its allowed entity.

Using the previous example, the AWS KMS key arn:aws:kms:us-east-1:XXXXXXXX9645:key/64d3e826-1a87-4cbc-95b8-fb386730a07 policy must have the following permissions attached:

{
    "Sid": "Allow use of the key",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::XXXXXXXX9645:role/AWSGlue-MyGlueCustomRole "
        ]
    },
    "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
    ],
    "Resource": "*"
}

Make sure that the AWS KMS key exists in the same Region as your AWS Glue job

To check which Region your AWS KMS key exists in, follow these steps:

1.    Open the AWS KMS console.

2.    Choose the Region that you're encountering the error in.

3.    In the navigation pane, choose either AWS managed keys or Customer managed keys. Then search for the AWS KMS key that you're using. You can also use the AWS CLI to search for the AWS KMS key. The following command returns a NotFoundException error if the key doesn't exist in the Region:

aws kms describe-key --key-id <Key-ID> --region <region-name>

Note: Be sure to replace <Key-ID> and <region-name> in this example with the Key ID and name of the Region you're using.

4.    Check the AWS CloudTrail API calls kms:Decrypt and kms:Encrypt to see if you observe any failures. Or, you can view all event logs by using kms.amazonaws.com as the event source.

Cross-account example

If the AWS Glue role and the AWS KMS key are in different accounts, use the customer managed key, and not the AWS managed key. AWS KMS keys can be used only when the principals are from the same account. You can't modify the key policy for AWS managed keys. When you access the AWS KMS key in the second account, make sure that permissions are present both on the source and the destination.

Assume you want to allow an AWS Identity and Access Management (IAM) user X in Account Y to an AWS KMS key in Account Z. In this case, the IAM policy must have the required AWS KMS actions. The same actions must be allowed from the AWS KMS key policy. For more information, see Allowing users in other accounts to use an AWS KMS key.

Here are the most common issues that occur when accessing an AWS KMS key from a cross account.

If the error message is caused by a service control policy (SCP), then check all the applied SCPs on your account. This helps you to identify the SCP that's restricting the kms:Decrypt action. Update the affecting SCP, and try your action again. For more information, see Listing the policies attached to a root, OU, or account and Updating an SCP. See the following example of this error:

User: arn:aws:iam::XXXXXXXX9645:user/KMSDemO is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:XXXXXXXX9645:key/64d3e826-1a87-4cbc-95b8-fb386730a07 with an explicit deny in a service control policy"

If the IAM entity has a permission boundary attached to it, then the boundary sets the maximum permissions that the entity has. Check the permissions boundary on the IAM principal to make sure that the required AWS KMS decrypt action is allowed. Then, try the action again. See the following example of this error:

User: arn:aws:iam::XXXXXXXX9645:user/KMSDemO is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:XXXXXXXX9645:key/64d3e826-1a87-4cbc-95b8-fb386730a07 because no permissions boundary allows the kms:Decrypt action"

If you're connecting to AWS KMS using a VPC endpoint, then check the applied VPC endpoint policy. Make sure that the kms:Decrypt action is allowed in the policy. See the following example of this error:

User: arn:aws:iam::XXXXXXXX9645:user/KMSDemO is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:XXXXXXXX9645:key/64d3e826-1a87-4cbc-95b8-fb386730a07 because no VPC endpoint policy allows the kms:Decrypt action"

---

Related information

Setting up encryption in AWS Glue

Key policies in AWS KMS