When I start my instance with encrypted volumes attached, the instance immediately stops with the error "client error on launch".

Last updated: 2020-09-25

I launched an Amazon Elastic Compute Cloud (Amazon EC2) instance that has encrypted volumes attached, but the instance doesn't start—the instance immediately goes from a pending state to a stopped state. I ran the AWS Command Line Interface (AWS CLI) command describe-instances and get this error:

....
"StateReason": {
  "Code": "Client.InternalError"
  "Message": "Client.InternalError: Client error on launch"
},
....

How can I resolve this?

Short description

This issue occurs with EC2 instances with encrypted volumes attached if:

  • The AWS Key Management Service (AWS KMS) or AWS Identity and Access Management (IAM) user launching the instances doesn't have the required permissions.
  • The KMS key usage is restricted by the SourceIp condition key.

The IAM user must have permission to AWS KMS to decrypt the AWS KMS key.

To allow access to decrypt a KMS key, you must use the key policy with IAM policies or grants. IAM policies alone aren't sufficient to allow access to a KMS key, but you can use them in combination with a KMS key's policy.

KMS keys by default grant access only to the root account. When EC2 full privilege is provided to an IAM user or role, AWS KMS permissions must explicitly grant access to the KMS keys policy.

Resolution

Create an IAM policy to allow the IAM principal to call AWS KMS APIs

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

1    Open the IAM console, choose Policies, and then choose Create policy.

2    Choose the JSON tab, and then copy and paste this policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:RevokeGrant",
                "kms:GenerateDataKey",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:DescribeKey",
                "kms:CreateGrant",
                "kms:ListGrants"
            ],
            "Resource": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
        }
    ]
}

3    Choose Review policy.

4    In Name, enter a name that is meaningful to you, and then choose Create policy.

5    Choose the policy that you created in step 4.

6    Choose the Policy usage tab, and then choose Attach.

7    In Name, choose the IAM entity that you want to grant permission to KMS key, and then choose Attach policy.

Grant the IAM principal explicit access to a KMS key

1    Open the AWS KMS console, and choose Customer managed keys.

2    In Key ID, choose you Key ID.

3    In Key users, choose Add.

4    In Name, choose the IAM user or role, and then choose Add.

Note: If you're using a custom key policy instead of the default key policy, the KMS key must explicitly grant the following permissions:

{
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                "arn:aws:iam::123456789012:role/MyRoleName"
                    "arn:aws:iam::123456789012:user/MyUserName",
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
            "arn:aws:iam::123456789012:role/MyRoleName"
                    "arn:aws:iam::123456789012:user/MyUserName",
                ]
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }

Use IP address condition

If you use AWS KMS to protect your data in an integrated service, then be careful when specifying the IP address condition operators, or when specifying the aws:SourceIp condition key in the same access policy statement. Attaching an encrypted Amazon Elastic Block Store (Amazon EBS) volume to an Amazon EC2 instance causes Amazon EC2 to send a request to AWS KMS to decrypt the volume's encrypted data key. This request comes from an IP address associated with the EC2 instance, not the user's IP address. This means that the decryption request is rejected if you have a SourceIp condition set, and the instance fails.

Use the kms:ViaService condition key. AWS KMS allows interactions from that service on your behalf. Be sure that the principals have permission to use the KMS key and integrated service. For more information, see kms:ViaService condition key limits.

Note: EC2 instances with logged-on users can't interact with this condition—only the service on your behalf can. This interaction is logged in AWS CloudTrail logs for you to review.

"userIdentity": {
  "sessionContext": {
    "sessionIssuer": {
      "accountId": "450822418798",
      "principalId": "450822418798:aws:ec2-infrastructure",
      "userName": "aws:ec2-infrastructure",
      "arn": "arn:aws:iam::450822418798:role/aws:ec2-infrastructure",
      "type": "Role"
     },
…
    "eventType": "AwsApiCall",
    "@log_group": "CloudTrail/AllRegionLogGroup",
    "awsRegion": "eu-west-1",
    "requestParameters": {
      "encryptionContext": {
        "aws:ebs:id": "vol-0ca158925aa9c1883"
      }    
},

In this example, the CloudTrail entry for an API call is made to AWS KMS. This is called on by Amazon EC2 infrastructure, not from a specific IP address. When you add a policy to a user that allows AWS KMS to interact with Amazon EC2, then the API call can complete.