Why can't my IAM user or role with full Amazon EC2 permissions start an EC2 instance?

Short description

Check to see if the instance has an Amazon Elastic Block Store (Amazon EBS) attached. If the Amazon EBS volume is encrypted using an AWS Key Management Service (AWS KMS) key, then there might be a permission issue. The IAM entity calling the StartInstances API action must have permissions to create a grant for the Amazon EC2 service. The grant allows Amazon EC2 to decrypt the AWS KMS key (KMS key).

Amazon EBS volumes send a GenerateDataKeyWithoutPlaintext API call request to AWS KMS that creates a new data key and encrypts it in the KMS key. The encrypted data key is sent back to the Amazon EBS volume, and then attached to the Amazon EC2 instance. For this scenario, the KMS key is present in the same AWS account for the EC2 instance and the KMS key.

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

1. To confirm why the Amazon EC2 instance is in a Stopped state, run the AWS CLI command similar to the following:

aws ec2 describe-instances --instance-id your-instance_ID --query "Reservations[*].Instances[*].StateReason"

Example output:

[
  [
    {
      "Message": "Client.InternalError: Client error on launch",
      "Code": "Client.InternalError"
    }
  ]
]

This error means that the root volume or additional attached volumes are encrypted. You don't have permission to access the AWS KMS key for decryption.

2. (Optional) Follow the instructions for filtering AWS CloudTrail events for the event name CreateGrant.

Example output:

"errorMessage": "User: arn:aws:iam::123456789012:user/test is not authorized to perform: kms:CreateGrant on resource: arn:aws:kms:eu-west-1:123456789012:key/8e3426b8-87b4-434c-ae74-8e63dadf354a"

This error means that the IAM entity doesn't have the CreateGrant permission for Amazon EC2 to decrypt the data key so the instance can't start.

Resolution

To find the key type, follow these steps:

1.    Open the Amazon EC2 console, and then choose Instances.

2.    In Instance ID, choose the Amazon EC2 instance ID, and then choose the Storage tab.

3.    In Volume ID, choose the volume ID for the encrypted volume.

4.    In KMS key ID, copy the key ID.

5.    Open the AWS KMS console in the same AWS Region.

6.    In AWS managed keys and Customer managed keys, paste the KMS key ID from step 4.

7.    Choose the Key ID.

8.    In General configuration under Description, note the KMS key type.

Attach an IAM policy to the IAM entity similar to the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:CreateGrant"
      ],
      "Resource": [
        "arn:aws:kms:eu-west-1:123456789012:key/8e3426b8-87b4-434c-ae74-8e63dadf354a"
      ],
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": true
        }
      }
    }
  ]
}

The kms:GrantIsForAWSResource condition key assures that the IAM entity is only able to create grants for the KMS key with AWS resources such as EC2 instances. This policy doesn't allow the IAM entity to create grants for another IAM entity.

(Optional) Allow the AWS account root user account full access to the KMS key similar to the following:

{
  "Sid": "Enable IAM User Permissions",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::123456789012:root"},
  "Action": "kms:*",
  "Resource": "*"
}

You can also add the IAM entity in the KMS key policy to allow the CreateGrant API action.

Note:

• It's a best practice to grant least privilege for only the permissions required to perform a task. For more information, see Grant least privilege.
• If your EC2 instances are created using Amazon EC2 Auto Scaling groups, then see Required AWS KMS key policy for use with encrypted volumes.

---

Related information

How EBS encryption works

Instance terminates immediately