Troubleshoot INCOMPATIBLE_CREATE issues due to missing instance profile permissions during Amazon RDS Custom instance creation

Amazon Relational Database Service (Amazon RDS) Custom brings the benefits of Amazon RDS to customers that can find it difficult to move to a fully managed service because of customizations needed with third-party applications. RDS Custom saves administrative time, is durable, and scales with your business. It does this by giving you tools to access and customize your database environment and operating system, helping you meet the requirements of legacy, custom, and packaged applications built on the Oracle and SQL Server database engines.

When you create an RDS Custom instance, it requires an instance profile to be provided as parameter during DB instance creation request. RDS custom uses it for a built-in automation within the DB instance, which evaluates the instance profile permissions. This can be impacted by AWS Identity and Access Management (IAM) policies, IAM permissions boundaries, resource-based policies, virtual private cloud (VPC) endpoint policies, and service control policies (SCPs). If any of these have explicit deny or incorrect permissions, an instance might go into an INCOMPATIBLE_CREATE state when provisioning.

In this post, we discuss how to determine the causes of an INCOMPATIBLE_CREATE state of an RDS Custom instance because of incorrect instance profile permissions.

Solution overview

Incorrect instance profile permissions can cause the RDS Custom instance to go into INCOMPATIBLE_CREATE state. To troubleshoot this error, you must have the correct IAM permissions. In the next sections, we demonstrate how to identify the cause of an incompatible create state by reviewing the Amazon RDS events and using it to identify any resource-specific permission or SCP permissions blocking instance creation.

We also share other issues that can cause the INCOMPATIBLE_CREATE error in RDS Custom due to missing instance profile permissions and steps that you can take to prevent or fix them.

Possible reasons for INCOMPATIBLE_CREATE

The following are the reasons an RDS Custom instance can go into an INCOMPATIBLE_CREATE state during creation due to missing instance profile permissions.

Instance profile missing required IAM policy permissions: It’s recommended to use an AWS Managed Policy. If you’re using a self-managed IAM policy, make sure it has all the required permissions if you are using it for RDS Custom SQL server or RDS Custom Oracle instance creation.
IAM permissions boundary: Permissions can be restricted because of an IAM permissions boundary attached to the instance profile role. Make sure the IAM permissions required by the RDS Custom instance aren’t restricted.
Resource-based policies: Permissions can be restricted using resource-based policies. Check if Amazon Simple Storage Service (Amazon S3), AWS KMS, or AWS Secrets Manager has policies attached to the resource that are blocking the access. For example, if an S3 bucket has a deny permission, the error signature is You can’t create the DB instance because of incompatible resources. Role attached to instance-profile missing required permission, The IAM instance profile role is missing the following permissions: S3:putObject action not allowed on S3 bucket.
VPC endpoint policy: Permissions can be restricted by a VPC endpoint policy if the VPC uses a VPC endpoint to reach the corresponding AWS service. Check if there are any VPC endpoint policies blocking access.
Service control policies (SCPs): SCPs are a type of organization policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for the IAM users and roles in your organization. SCPs can block access to any resources with an explicit deny or not having explicit allow. If SCPs uses condition keys, such as aws:RequestedRegion to restrict AWS Regions where actions are allowed, IAM policy simulator always returns deny even if permissions are allowed in the specific Region. In these kinds of issues, reach out to AWS Support for further assistance. You can confirm from your end if the SCP policy is included in your account.

You can check if your account has an SCP policy from the AWS Organizations console. In the console, go to Policies and check if the SCP policy is enabled. You can also check using the following AWS CLI command:

$ aws organizations list-policies --filter SERVICE_CONTROL_POLICY

For more troubleshooting steps on errors related to SCPs, see How SCPs work with Deny.

In this blog post we demonstrate a scenario where the RDS Custom for SQL Server instance creation fails due to a resource-based policy in AWS KMS.

Prerequisites

In the upcoming sections, we demonstrate different scenarios that can cause the INCOMPATIBLE_CREATE error in RDS Custom. Should you want to follow along, you must:

Grant required permissions to your IAM principal.
Make sure to supply a networking configuration that RDS Custom can use to access other AWS services. For specific requirements, see Configuring your VPC manually , Configure your VPC security group and Configure endpoints for dependent AWS services.
Make sure you have an AWS Key Management Service (AWS KMS) symmetric encryption key.
Ensure the required IAM role with the managed policy and instance profile is created and included when launching the RDS Custom instance.
Install or update the latest version of the AWS Command Line Interface (AWS CLI).

Walkthrough

For this demonstration, we have an explicit deny on the AWS KMS key policy, causing INCOMPATIBLE_CREATE during the RDS Custom instance creation.

Create an IAM role with the following trust relationship and the managed policy. We named it AWSRDSCustomtestrole.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Create the Instance profile using AWS CLI.

aws iam create-instance-profile \
--instance-profile-name AWSRDSCustomtestrole

Add the role to the instance profile.

aws iam add-role-to-instance-profile \
--instance-profile-name AWSRDSCustomtestrole \
--role-name AWSRDSCustomtestrole

Create a KMS key and make sure that your symmetric encryption key policy grants access to the kms:Decrypt and kms:GenerateDataKey operations to the IAM role in your IAM instance profile.
Note: We add a deny permission to access the key for the IAM role to force the error.

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Deny use of Key",
            "Effect": "Deny",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:role/AWSRDSCustomtestrole"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}

Create an RDS Custom instance and use the instance profile and KMS key created previously.
After initiating the instance creation, the RDS Custom instance goes into INCOMPATIBLE_CREATE state
To troubleshoot the issue, you can review the logs and events section in the AWS Management Console for Amazon RDS. The following output is from the instance’s creation.
You can't create the DB instance because of incompatible resources. Role attached to instance-profile missing required permission, The IAM instance profile role is missing the following permissions: kms:Decrypt action not allowed on kms key [arn:aws:kms:us-east-1:123456789:key/key-id]., kms:GenerateDataKey action not allowed on kms key [arn:aws:kms:us-east-1:123456789:key/key-id].

Modify the KMS key policy to allow the required permissions:

{
    "Version": "2012-10-17",
    "Id": "key-consolepolicy-3",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow use of Key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:role/AWSRDSCustomtestrole"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}

Create an RDS Custom instance and use the instance profile and KMS key created previously.
The instance is created without INCOMPATIBLE_CREATE.

Clean up

To avoid ongoing charges, delete the resources you created with this solution:

Conclusion

In this post, we reviewed the factors that can cause RDS Custom to go into INCOMPATIBLE_CREATE state. We also highlighted some of the other factors that you consider for a smooth deployment of your RDS Custom instance.

If you have any questions or concerns, leave them in the comment section.

About the authors

Nirupam Datta is a Senior Cloud Support DBE at AWS and has been with AWS for over 4 years. With over 12 years of experience in database engineering and infra-architecture, Nirupam is a subject matter expert in the Amazon RDS core systems and Amazon RDS for SQL Server and Amazon Aurora MySQL. He provides technical assistance to customers, guiding them to migrate, optimize, and navigate their journey in the AWS Cloud.

Noorul Mahajabeen Mustafa is a Cloud Support DBE working with AWS for 2.5 years. With over 5 years of database experience, Noorul is also a subject matter expert in Amazon RDS SQL server, AWS Database Migration Service (AWS DMS) and Amazon Aurora MySQL. She works with customers providing exceptional technical assistance in database migration, RDS infrastructure, monitoring, and security related scenarios.

Dipin Sahadevan is a Cloud Support Engineer working with AWS for over 2 years. Dipin is also a subject matter expert in Amazon RDS for SQL Server. He has 15 years of experience working with relational databases. At AWS he works with external customers to handle a variety of scenarios, such as troubleshooting RDS infrastructure and, authoring and improving internal documentation.

AWS Database Blog