AWS Database Blog

Automatically scale Amazon RDS storage using Amazon CloudWatch and AWS Lambda

Customer databases are constantly growing. When you increase storage for a database in Amazon Relational Database Service (Amazon RDS), you can’t make further storage modifications for either 6 hours or until storage optimization has completed on the instance, whichever is longer. Amazon RDS storage autoscaling increases storage by 10 GiB, 10% of currently allocated storage, or the predicted storage growth for the next 7 hours based on the past hour, whichever is greatest. Amazon RDS storage autoscaling is suitable for the majority of databases and usage patterns. However, if you regularly perform large data loads and storage autoscaling doesn’t provide enough space, the database might remain in the storage full state for several hours. This can harm the database and also prevent writes and connections when the storage is full.

In this post, we show you an automated solution using Amazon CloudWatch and AWS Lambda to choose how much your database scales storage by when it’s running out of space. The more you scale storage by, the less likely it is to run out of space again during the 6-hour minimum storage optimization period. Scaling storage by too much results in excessive free space, so make sure you select a scaling percentage that best fits your use case and data loading patterns. This solution does not replace storage autoscaling, but is rather placed behind storage autoscaling for customers who need to scale their database by more than the default increment.

Overview of the solution

In this solution, we use CloudWatch to monitor an RDS database’s free storage space, and send a notification to Amazon Simple Notification Service (Amazon SNS) when the free storage space is below your defined amount. Amazon SNS then invokes a Lambda function with an event that includes the name of the database under the free storage space alarm. The Lambda function gets the amount of storage in this database, adds a user-specified percentage of storage, and modifies the database with this new storage value. When the database grows, the CloudWatch alarm will return to OK status. If the database exceeds your preset threshold again, the cycle repeats because the CloudWatch alarm will go back into an alarm state and scale up storage for the database again.

This solution doesn’t allow you to make storage modifications to an RDS database within the 6-hour storage optimization period. It simply allows you to choose a greater scaling percentage, such as 20%, 30%, or 40%, to reduce the likelihood of running out of space again during the 6-hour period.

The following diagram illustrates the solution architecture.

In the following sections, we discuss how to set up all the resources necessary for this Amazon RDS storage scaling solution.

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account
  • An existing RDS database to monitor and scale as necessary

Create solution resources with AWS CloudFormation

We use an AWS CloudFormation template to set up the solution resources. The CloudFormation stack creates a Lambda function, Lambda AWS Identity and Access Management (IAM) execution role, SNS topic, and SNS subscription. Follow the instructions below, or alternatively watch the video walkthrough at the end of this post.

  1. Download the CloudFormation template (YAML file).
  2. On the AWS CloudFormation console, choose the Region you would like to deploy this solution to.
  3. Create a stack with new resources.
  4. Upload the YAML file.
  5. Enter a name for the stack name, and optionally choose an IAM role for the stack.

If no IAM role is selected, AWS CloudFormation uses the credentials from the current user.

Create a CloudWatch alarm

After you create the CloudFormation stack, you can create a CloudWatch alarm:

  1. On the CloudWatch console, choose Alarms in the navigation pane.
  2. Choose Create alarm.
  3. Enter the database identifier in the search bar, and choose RDS > DBInstanceIdentifier.
  4. Select FreeStorageSpace as the metric and choose Select Metric.
  5. Select the conditions for the CloudWatch alarm to trigger, specifically how many bytes of free storage the database should scale up to.
    In order to have the CloudWatch alarm trigger before storage autoscaling automatically triggers, we suggest setting the alarm to trigger at 15% of the total storage space. If you have a 100 GB database, 15% of that (15 GB, or 15,000,000,000 bytes) would be when the alarm should trigger the scaling action.

    The Lambda function will automatically scale up the CloudWatch alarm threshold by the same percentage as the database storage, so that if a database grows from 100 GB to 200 GB, the alarm will trigger at 30 GB of free storage space instead of 15 GB.

    By setting the free storage space to 15% of the total database storage, it allows us to keep Amazon RDS storage autoscaling enabled at 10% of free space as a fallback in case the solution fails.

    If you keep storage autoscaling enabled (recommended), be sure to increase the value for the Amazon RDS maximum storage threshold, because your RDS database will never be able to scale beyond that amount even with this automated solution.

  6. In the next step, select Select an existing SNS topic and enter the topic RDS-SNS-Storage-Scaler to notify when the CloudWatch alarm is in an alarm state.

  7. Enter any name for the CloudWatch alarm and create it.

Lambda function code

With the CloudWatch alarm now created, whenever the free storage reaches the amount you selected, it will send a notification to the SNS topic, which will invoke the Lambda function. The Lambda function then increases the storage of the RDS database. Let’s look at part of the Lambda code in detail:

import sys, botocore, boto3, json, math
def lambda_handler(event, context):
ScalingUnder1000GB = 1.2
Scaling1000GBto5000GB = 1.15
ScalingOver5000GB = 1.10

In lines 3–5, we define three variables for the scaling percentage that you can modify based on your preferences:

  • ScalingUnder1000GB = 1.2
  • Scaing1000GBto5000GB = 1.15
  • ScalingOver5000GB = 1.10

By default, this Lambda function scales storage for databases under 1,000 GB by 20%, databases between 1,000–5,000 GB by 15%, and databases over 5,000 GB by 10%. However, you can modify these values in the Lambda function to scale databases by smaller or greater amounts. The minimum amount to scale Amazon RDS storage by is 10%.

The CloudFormation template we provided is sufficient for single-Region deployments. If you’re deploying this solution to sequential Regions, you need to use the following CloudFormation template, which creates only the Lambda function and SNS topic. You don’t need to recreate the IAM roles for sequential Regions because IAM is a global service.

Clean up

If you would like to decommission this solution to no longer scale up storage for RDS instances, you can delete the CloudFormation stack, which deletes the Lambda function, SNS topic, Lambda execution role, and policy. Finally, you must also delete the CloudWatch alarm you created.

Conclusion

In this post, we showed you a solution that uses a CloudWatch alarm that continuously monitors free storage space for an RDS database, and sends a notification to an SNS topic when the database is running out of storage. The SNS topic then invokes a Lambda function to increase storage of the RDS database by an amount you choose, and you can continuously change the scaling amount. You can use the same single SNS topic and Lambda function for monitoring and scaling several databases by creating several CloudWatch alarms. Download the CloudFormation template to get started with this solution, and feel free to add comments to this post with any questions or feedback.

About the author

Yassin Abouel Seoud is an Enterprise Support Lead at Amazon Web Services. He works with customers on improving their workloads’ performance and resiliency, and resolving blockers. Prior to AWS, Yassin completed his Bachelor of Engineering in Mechanical Engineering at McGill University, and worked on expanding Alexa’s understanding and answering internationally.