How can I rotate an AWS Secrets Manager secret in a private VPC?

Last updated: 2019-11-06

I tried to rotate an AWS Secrets Manager secret for an AWS service in a private Amazon VPC. However, the operation failed and Amazon CloudWatch Logs show that the AWS Lambda task timed out.  

Short Description

Secrets Manager can't rotate secrets for AWS services running in Amazon VPC private subnets because they don't have internet access.  

Resolution

Follow these instructions to configure an Amazon VPC interface endpoint to access your Secrets Manager Lambda function and private Amazon RDS instance. In the following example, a private Aurora RDS instance in Amazon VPC named vpc-0abb11f5a28a8abe7 is used.

Important: Before you begin, be sure you have installed and configured the AWS Command Line Interface (AWS CLI).

Create security groups for the Secrets Manager VPC endpoint, Amazon RDS instance, and the Lambda rotation function

Follow the instructions for creating a security group similar to the following AWS CLI commands:

1.    Security group for Secrets Manager Amazon VPC endpoint:

Note: Replace vpc-id vpc-0abb11f5a28a8abe7 with your VPC ID.

$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name SMVPCEndpointSG --description "secretsmanager VPCEndpoint SG"
{
    "GroupId": "sg-0bacf4bbed67e4df5"
}

2.    Security group for Lambda rotation function:  

$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name LambdaFunctionSG --description "Lambda Rotation Function SG"
{
    "GroupId": "sg-0aed64f81acc4c037"
}

3.    (Optional) Create a security group for the Amazon RDS instance:

Note: This step is needed if your RDS instance uses only the default security group.  

$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name RDSInstanceSG --description "RDS Instance SG"
{
    "GroupId": "sg-0d06795dffa052563"
}

Create an Amazon VPC interface endpoint for the Secrets Manager service and associate it with a security group

Follow the instructions for creating an interface endpoint similar to the following:

Note: Replace your-region with your AWS Region.  

$ aws ec2 create-vpc-endpoint --vpc-id  vpc-0abb11f5a28a8abe7 --vpc-endpoint-type Interface \
--service-name com.amazonaws.your-region.secretsmanager  --subnet-ids subnet-076c28105d486f3bd subnet-0af00c796ccdc725f \
--security-group-ids sg-0bacf4bbed67e4df5

Important: Your Amazon VPC must have DNS hostnames and DNS resolution attributes enabled. For more information, see Viewing and Updating DNS Support for Your VPC.

Add rules to Amazon VPC endpoint and Amazon RDS instance security groups

1.    Configure the security group rules for the Amazon VPC endpoint to allow inbound traffic from the security group of the Lambda function on port 443:

$ aws ec2 authorize-security-group-ingress --group-id sg-0bacf4bbed67e4df5 --protocol tcp --port 443 --source-group sg-0aed64f81acc4c037

2.    The security group for the Amazon RDS instance allows inbound connection from the security group of the Lambda function on port 3306:

Note: Replace your-rds-security-group with your security group (either an existing security group or the optional RDS Instance SG).  

$ aws ec2 authorize-security-group-ingress --group-id your-rds-security-group --protocol tcp --port 3306 --source-group sg-0aed64f81acc4c037

Attach security groups to AWS resources

1.    If you created the optional RDS Instance SG, modify the RDS instance configuration, similar to the following:

Note: Replace your-existing-rds-security-groups with the group or groups attached to the RDS instance.

$ aws rds modify-db-instance --db-instance-identifier your-rds-instance --vpc-security-group-ids sg-0d06795dffa052563 your-existing-rds-security-groups

2.    Follow the instructions to update the Lambda function configuration similar to the following:

$ aws lambda update-function-configuration --function-name your-lambda-function \
--vpc-config SubnetIds=subnet-076c28105d486f3bd,subnet-0af00c796ccdc725f,SecurityGroupIds=sg-0aed64f81acc4c037

Verify that Secrets Manager is able to rotate the secret

1.    Follow the instructions for rotating the Secrets Manager secret similar to the following:

Note: Replace your-secret with your Secrets Manager secret.  

$ aws secretsmanager rotate-secret --secret-id your-secret

Secrets Manager attempts to retry the previous rotation.

Note: Because previous attempts to rotate the password are unsuccessful, you receive output similar to the following:

An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.

2.    Monitor the function in the AWS Lambda Console. If the rotation is successful, then the Amazon CloudWatch log streams contain an entry similar to the following:  

[INFO] 2019-10-22T07:59:32.627Z 96179023-5b67-4e98-a057-885f68bc69f2 finishSecret: Successfully set AWSCURRENT stage to version 175b5e38-341f-4cd0-8c58-2b1e49769642 for secret arn:aws:secretsmanager:your-region:your-account:secret:your-secret

3.    Retrieve the Secrets Manager secret to confirm that it rotated successfully, similar to the following:

Note: replace your-secret-arn with your Secrets Manager secret ARN.  

aws secretsmanager get-secret-value --secret-id your-secret-arn

Note: The Secrets Manager rotation function runs asynchronously in the background, and can take several minutes to complete.