Automate Amazon RDS credential rotation with AWS Secrets Manager for primary instances with read replicas

The Amazon Relational Database Service (Amazon RDS) and AWS Secrets Manager integration offers robust credential management for database instances. When using Secrets Manager to manage your master user passwords, you cannot create new read replicas for your database instance. This applies to all DB engines except Amazon RDS for SQL Server, potentially impacting your organization’s ability to efficiently scale its read operations while maintaining secure credential practices.

In this post, we present a solution that automates the process of rotating passwords for a primary instance with read replicas while maintaining secure credential management practices. This approach allows you to take advantage of the benefits of both read scaling and automated credential rotation.

Solution overview

For this post, we use an AWS Lambda function that orchestrates the rotation password for primary instances with read replicas and manages their credentials of the source database’s Secrets Manager integration. Here’s a high-level overview of the process:

Secrets Manager automatically triggers the Lambda function based on the configured rotation schedule.
This initiates the secure credential rotation workflow
The Lambda function generates new secure credentials
These credentials are configured to work with both Primary database instance and associated read replica(s)
The newly generated credentials are securely stored in AWS Secrets Manager
The entire process repeats automatically based on your defined rotation schedule ensuring regular password updates without manual intervention

This approach ensures that RDS primary instances and their read replicas maintain synchronized credentials, with all password management handled securely through AWS Secrets Manager.

The following diagram illustrates the solution architecture.

Set up an RDS primary instance and create a read replica

In this section we create the Lambda function that will rotate the password for primary database instance with read replica automatically. Complete the following steps to configure an RDS instance and read replica:

On the Amazon RDS console, choose Databases in the navigation pane.
Choose Create database.
For Choose a database creation method, choose Standard create.
Under Engine options, for Engine type, choose PostgreSQL.
Under Credentials Settings, for Credentials management, select Self managed. (If you choose “Managed in AWS Secrets Manager” then we cannot create read replica for this instance, so selecting Self managed is important)
Fill out the other parameters based on your scenario and create the instance. If you created read replicas to your instance you can move on to the next section. If you haven’t, create one (or more) read replica(s) to your primary instance.

Create the Lambda function

In this section we create the Lambda function that will automate the rotation of password in secret manager for primary RDS instances with read replica(s). Complete the following steps to create your Lambda function:

On the Lambda console, choose Functions in the navigation pane.
Choose Create function.
For Function name, enter a name for your function.
For Runtime, choose Python 3.13.
For Execution role, select Create a new role with basic Lambda permissions. (Later in this guide, we will configure additional IAM permissions for RDS and AWS Secrets Manager in the Lambda function’s role)
Choose Create function.

On the Code tab of the Lambda function, enter the following Python code:

import boto3
import json
import string
import random
import os
from botocore.exceptions import ClientError

def lambda_handler(event, context):
    """Secrets Manager RDS Password Rotation Lambda function"""
    print(f"Starting password rotation with event: {event}")
    arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']

    # Setup the clients
    rds_client = boto3.client('rds')
    secrets_client = boto3.client('secretsmanager')

    # Make sure the version is staged correctly
    print("Checking secret metadata and rotation status...")
    metadata = secrets_client.describe_secret(SecretId=arn)
    if not metadata['RotationEnabled']:
        print(f"ERROR: Secret {arn} is not enabled for rotation")
        raise ValueError(f"Secret {arn} is not enabled for rotation")
    
    versions = metadata['VersionIdsToStages']
    print(f"Current version stages: {versions}")

    if token not in versions:
        print(f"ERROR: Version {token} not found in version stages")
        raise ValueError(
            f"Secret version {token} has no stage for rotation of secret {arn}")

    if "AWSCURRENT" in versions[token]:
        print("Secret version is already marked as AWSCURRENT. Exiting.")
        return
    elif "AWSPENDING" not in versions[token]:
        print(f"ERROR: Version {token} not in AWSPENDING stage")
        raise ValueError(
            f"Secret version {token} not set as AWSPENDING for secret {arn}")
   
    if step == "createSecret":
        create_secret(secrets_client, arn, token, rds_client)
    elif step == "setSecret":
        wait_for_db_availability(rds_client, secrets_client, arn, token)
    elif step == "testSecret":
        check_rds_status(rds_client, secrets_client, arn, token)
    elif step == "finishSecret":
        finish_secret(secrets_client, arn, token)
    else:
        raise ValueError("Invalid step parameter")


def create_secret(secrets_client, arn, token, rds_client):
    """Generate a new secret value and put it to AWSPENDING"""
    print(f"Starting create_secret step for secret: {arn}")

    new_password = get_random_password(secrets_client)
    print("Successfully generated new password")

    try:
        db_instance_id = os.environ['DB_IDENTIFIER']
        print(f"Getting master username for DB instance: {db_instance_id}")

        response = rds_client.describe_db_instances(
            DBInstanceIdentifier=db_instance_id
        )
        master_username = response['DBInstances'][0]['MasterUsername']
        print(f"Retrieved master username: {master_username}")

        secret_dict = {
            'username': master_username,
            'password': new_password
        }
        print("Storing new secret in AWSPENDING stage...")
        secrets_client.put_secret_value(
            SecretId=arn,
            ClientRequestToken=token,
            SecretString=json.dumps(secret_dict),
            VersionStages=['AWSPENDING']
        )
        print("Successfully stored new secret value")

    except ClientError as e:
        print(f"ERROR in create_secret: {str(e)}")
        raise ValueError(f"Error creating secret: {str(e)}")

    try:
        print(f"Updating password in RDS instance: {db_instance_id}")
        rds_client.modify_db_instance(
            DBInstanceIdentifier=db_instance_id,
            MasterUserPassword=new_password,
            ApplyImmediately=True
        )
        print(
            f"Successfully updated password for RDS instance {db_instance_id}")

    except ClientError as e:
        print(f"ERROR updating database password: {str(e)}")
        raise ValueError(f"Error updating database: {str(e)}")


def wait_for_db_availability(rds_client, secrets_client, arn, token):
    print(f"Starting set_secret step for secret: {arn}")
    try:
        db_instance_id = os.environ['DB_IDENTIFIER']
        print(
            f"Waiting for DB instance {db_instance_id} to become available...")

        waiter = rds_client.get_waiter('db_instance_available')
        waiter.wait(
            DBInstanceIdentifier=db_instance_id,
            WaiterConfig={
                'Delay': 30,
                'MaxAttempts': 10
            }
        )
        print(f"DB instance {db_instance_id} is now available")

    except ClientError as e:
        print(f"ERROR in checking for Database availability: {str(e)}")
        raise ValueError(f"Error for Database availability: {str(e)}")


def check_rds_status(rds_client, secrets_client, arn, token):
    print(f"Starting check_rds_status step for secret: {arn}")
    try:
        db_instance_id = os.environ['DB_IDENTIFIER']
        print(f"Checking status of DB instance: {db_instance_id}")

        response = rds_client.describe_db_instances(
            DBInstanceIdentifier=db_instance_id
        )
        status = response['DBInstances'][0]['DBInstanceStatus']
        print(f"Current DB instance status: {status}")

        if status != 'available':
            print(
                f"ERROR: Database is not available. Current status: {status}")
            raise ValueError("Database is not available")

        print("Successfully verified database availability")

    except ClientError as e:
        print(f"ERROR in checking rds status: {str(e)}")
        raise ValueError(f"Error checking rds status: {str(e)}")


def finish_secret(secrets_client, arn, token):
    print(f"Starting finish_secret step for secret: {arn}")
    try:
        print("Retrieving secret metadata...")
        metadata = secrets_client.describe_secret(SecretId=arn)
        current_version = None

        for version in metadata["VersionIdsToStages"]:
            if "AWSCURRENT" in metadata["VersionIdsToStages"][version]:
                current_version = version
                break

        print(f"Current version: {current_version}")
        print(
            f"Updating secret version stage from {current_version} to {token}")

        secrets_client.update_secret_version_stage(
            SecretId=arn,
            VersionStage="AWSCURRENT",
            MoveToVersionId=token,
            RemoveFromVersionId=current_version
        )
        print("Successfully updated secret version to AWSCURRENT")

    except ClientError as e:
        print(f"ERROR in finish_secret: {str(e)}")
        raise ValueError(f"Error finishing secret: {str(e)}")

def get_environment_bool(variable_name, default_value):
    """Loads the environment variable and converts it to the boolean.

    Args:
        variable_name (string): Name of environment variable

        default_value (bool): The result will fallback to the default_value when the environment variable with the given name doesn't exist.

    Returns:
        bool: True when the content of environment variable contains either 'true', '1', 'y' or 'yes'
    """
    variable = os.environ.get(variable_name, str(default_value))
    return variable.lower() in ['true', '1', 'y', 'yes']


# Generate random password
def get_random_password(rds_client):
    """ Generates a random new password. Generator loads parameters that affects the content of the resulting password from the environment
    variables. When environment variable is missing sensible defaults are chosen.

    Supported environment variables:
        - EXCLUDE_CHARACTERS
        - PASSWORD_LENGTH
        - EXCLUDE_NUMBERS
        - EXCLUDE_PUNCTUATION
        - EXCLUDE_UPPERCASE
        - EXCLUDE_LOWERCASE
        - REQUIRE_EACH_INCLUDED_TYPE

    Args:
        rds_client (client): The secrets manager service client

    Returns:
        string: The randomly generated password.
    """
    passwd = rds_client.get_random_password(
        ExcludeCharacters=os.environ.get('EXCLUDE_CHARACTERS', ':/@"\'\\'),
        PasswordLength=int(os.environ.get('PASSWORD_LENGTH', 32)),
        ExcludeNumbers=get_environment_bool('EXCLUDE_NUMBERS', False),
        ExcludePunctuation=get_environment_bool('EXCLUDE_PUNCTUATION', False),
        ExcludeUppercase=get_environment_bool('EXCLUDE_UPPERCASE', False),
        ExcludeLowercase=get_environment_bool('EXCLUDE_LOWERCASE', False),
        RequireEachIncludedType=get_environment_bool(
            'REQUIRE_EACH_INCLUDED_TYPE', True)
    )
    return passwd['RandomPassword']

Create a Secrets Manager secret

Complete the following steps to store a new secret for your RDS instance in Secrets Manager:

On the Secrets Manager console, choose Secrets in the navigation pane.
Choose Store a new secret.
Select Other type of secret.
Enter user or admin credentials under Key/value pairs. add the username in key and password in the value)
Choose Next.
Enter a name for the secret.
Choose Next.
Choose Automatic rotation.
Fill in the rotation schedule.
Under Rotation function, choose the Lambda function you previously deployed.
Choose Next.
Review the settings and choose Store.

For this post, we set the rotation schedule as 4 hours. This means that every 4 hours, the password will be rotated in Secrets Manager, and we can connect to the RDS instance and its read replica using the new password.

Set up IAM permissions for the Lambda function

In this next step, you create an AWS Identity and Access Management (IAM) policy as the role for the Lambda function. The policy gives the Lambda function the required permissions to perform a secret rotation for users in the RDS DB instance.

Complete the following steps to configure the IAM role for the Lambda function:

On the Lambda console, navigate to your function.
In the Configuration section of the function overview page, choose Permissions.
Under Execution role, choose the link to the Lambda role name to view the role on the IAM console.
Under Permissions policies, choose Add permissions and Create inline policy.
For Policy editor, choose JSON.

Replace the JSON in the text editor with the secret rotation policy. Replace the Amazon Resource Name (ARN) for the secret being rotated and the ARN for the RDS instance in the IAM policy with values from your AWS account:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "rds:DescribeDBInstances",
                "rds:ModifyDBInstance"
            ],
            "Resource": "arn:aws:rds:us-east-1:xxxxxxxxx:db:test-primary"
        },
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:UpdateSecretVersionStage"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-1:XXXXXXXXX:secret:password-roatation-for-test-primary-replica-b1Rdt0"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "secretsmanager:GetRandomPassword",
            "Resource": "*"
        }
    ]
}

Choose Next.
Enter a policy name and choose Create policy.

Set up Lambda invocation rights for Secrets Manager

Complete the following steps to update the Lambda function permissions:

On the Lambda console, navigate to your function.
Under Resource-based policy statements, choose Add permissions.
Select AWS service.
For Service, choose Secrets Manager.
For Statement ID, enter a statement ID of your choosing.
For Action, choose lambda:InvokeFunction.
Choose Save.

Configure environment variables for the Lambda function

Complete the following steps to configure your function’s environment variables:

On the function overview page of the Lambda function, choose Configuration.
Choose Environment variables and choose Edit.
Choose Add environment variable.
Enter DB_IDENTIFIER for the key.
Enter the RDS instance name for the value. For this demo, our instance name is test-primary.
Choose Save.

Configure timeout settings for the Lambda function

On the function overview page of the Lambda function, choose General configuration.
Choose Edit and modify the Timeout to ~1 min.
For this demo, I have added 1min 3 sec.
Choose Save.

Verify secret password rotation

We recommend testing it in a non-production environment first. Create a test RDS instance with a read replica, then use the Lambda function to rotate the password. Verify that the password is created and rotated successfully and that its credentials are managed independently in Secrets Manager.

To verify the password rotation, complete the following steps:

On the Secrets Manager console, and navigate to the secret you created for your RDS credentials.
In the secret details, review the Secret value You should see that the password has been updated to a new value. The last rotation date should reflect the most recent rotation event.
Check the rotation history of the secret. This can be found in the Rotation configuration section of the secret details. Verify that rotations are occurring at the scheduled intervals you set up.
To further confirm the rotation’s success, attempt to connect to your RDS instance using the new credentials stored in the Secrets Manager. If you can successfully connect, this indicates that the rotation process has updated the database credentials correctly.
Review the Amazon CloudWatch logs for your Lambda function. Look for any error messages or unexpected behavior during the rotation process. Successful rotations should be logged without any errors.

Limitation

This solution is designed to manage password rotation for a single RDS instance with read replicas within an AWS account. While the solution can be used for multiple databases, each database would require:

A separate Lambda function deployment
Its own Secrets Manager secret
Corresponding IAM permissions for the Lambda function

Clean up

To avoid incurring unnecessary charges and to clean up the resources used in this solution, follow these steps:

Delete the Lambda function:
1. On the Lambda console, choose Functions in the navigation pane.
2. Select the function you created for this solution, and on the Actions menu, choose Delete.
3. Confirm the deletion when prompted.
Delete the IAM role:
1. On the IAM console, choose Roles in the navigation pane.
2. Select the role created for the Lambda function and choose Delete.
3. Confirm the deletion when prompted.
Delete the Secrets Manager secret:
1. On the Secrets Manager console, choose Secrets in the navigation pane.
2. Select the secret you created for storing your Amazon RDS credentials, and on the Actions menu, choose Delete secret.
If you created RDS instances specifically for testing this solution, you may want to delete
Review and clean up the CloudWatch logs:
1. On the CloudWatch console, choose Log groups in the navigation pane.
2. Locate and delete any log groups associated with your Lambda function.

Conclusion

In this post, showed how you can use AWS Lambda to create a solution that automates password rotation in Secrets Manager on Amazon RDS databases with read replicas. This approach allows you to maintain secure credential management practices while still benefiting from the read scaling capabilities of Amazon RDS.

If you have any questions or suggestions about this post, leave a comment.

About the authors

Lavanya Salokye is a Cloud Support DBE at Amazon Web Services, specializing in troubleshooting and optimizing cloud-based solutions. Her role involves assisting customers with AWS services, diagnosing technical issues, data migration and providing expert guidance on best practices for cloud architecture, security, and performance.

Lavu Pawar is a Delivery Consultant with AWS Web Services, supporting customers in designing, building, and deploying highly available and scalable solutions in the AWS Cloud. In his free time, he enjoys reading and spending quality time with his family and friends.

Sudhakar Darse is a Database Specialist Solutions Architect at Amazon Web Services. He works with AWS customers to provide guidance and technical assistance on database services, helping them with database migrations to the AWS Cloud and improving the value of their solutions when using AWS.

Select your cookie preferences

AWS Database Blog