A serverless solution to monitor the storage of your Amazon DynamoDB tables
If you use Amazon DynamoDB as your application’s NoSQL database service, you might want to track how much storage your DynamoDB tables are using. DynamoDB publishes its service metrics to Amazon CloudWatch. CloudWatch helps you monitor and analyze those metrics, set alarms, and react automatically to changes in your AWS resources. Currently, DynamoDB sends many useful metrics to CloudWatch, including
It’s important that you analyze the storage usage of your DynamoDB tables to get a historical overview of how much storage your DynamoDB workload uses, and to help control storage costs. For example, you might want to use Time to Live (TTL) to expire unnecessary or stale data. You could then use streams to archive that data to Amazon S3 or Amazon Glacier. In this post, I explain how you can monitor your DynamoDB table’s use of storage.
DynamoDB updates your table’s storage use every six hours. You can get that information by using the DynamoDB DescribeTable API. After determining the table’s size, you can create a custom CloudWatch metric to push the data to CloudWatch. In this solution, I show how to create a one-click deployment architecture to monitor your DynamoDB table’s storage use by using AWS CloudFormation, which you will use to implement an infrastructure as code model.
The following Python script and AWS CloudFormation template are also available in this GitHub repository.Here is how the process works, as illustrated in the preceding diagram. (Be sure to deploy this solution in every AWS Region where you have DynamoDB workloads and to continue monitoring the storage use of your DynamoDB tables.)
- Amazon CloudWatch Events executes an AWS Lambda function on a scheduled basis.
- The Lambda function checks the size of all DynamoDB tables in a Region by using the
- The Lambda function stores the table size information in a custom CloudWatch metric.
- Lambda uses a custom AWS Identity and Access Management (IAM) role created by an AWS CloudFormation template to access DynamoDB and CloudWatch.
- AWS CloudFormation deploys the whole solution, including creating the IAM role and the Lambda function, and scheduling the event.
Monitor your DynamoDB table’s storage use
Follow these steps to monitor your DynamoDB table’s storage use:
- Create a file called c
ddbstoragemon.py, and paste the following code into the file. This example code calls the
DescribeTableAPI for all the tables in a particular AWS Region, reads
TableSizeBytesfrom the API response, and puts that information in a custom CloudWatch metric called
- Zip the
ddbstoragemon.pyfile, and save it as
- Upload the .zip file to an Amazon S3
Deploy your code using AWS CloudFormation
Now that the code is in Amazon S3, deploy the entire solution by using AWS CloudFormation:
- Copy the CloudFormation template from GitHub, and save the file as
ddbStoreMonCF.json. The code creates the following:
- An IAM role to access the DynamoDB
DescribeTableAPI to get the DynamoDB table size and store that information in CloudWatch.
- A Lambda function using the Python script that you stored in Amazon S3.
- A CloudWatch Events rule to trigger the Lambda function every six hours.
- An IAM role to access the DynamoDB
- Sign in to the AWS Management Console, and navigate to the AWS CloudFormation console.
- Choose Create Stack.
- On the Select Template page, choose Upload a template to Amazon S3. Choose Browse, and then choose the
ddbStoreMonCF.jsontemplate that you created earlier.
- Choose Next. Type the Stack name, your Amazon S3 BucketName, and the FileName of the .zip file that you uploaded earlier to S3.
- Choose Next. Keep the default settings on the Options page, and choose Next.
- Select the I acknowledge that AWS CloudFormation might create IAM resources check box, and choose Create.
AWS CloudFormation creates the resources based on the template definition. After a few minutes, you should see that CloudFormation has created the stack successfully, as shown in the following screenshot.
Test the configuration
The CloudWatch rule that was created by the AWS CloudFormation template will trigger every six hours and store the associated metrics in CloudWatch. However, let’s do a quick test to confirm that the configuration works:
- Navigate to the CloudWatch console, and in the navigation pane, choose Rules.
- Search for the rule that has the AWS CloudFormation stack ddbstoragemon in its name, and choose it.
- The rule has been scheduled to trigger a Lambda function every six hours. Choose the Resource name to open the Lambda function created by AWS CloudFormation.
- In the AWS Lambda console, choose Test. In the Configure test event window, type an event name, accept the default value, and choose Create.
- Choose Test to run the function.
The Lambda function checks all of your DynamoDB table’s storage metrics for that AWS Region and puts that information in a custom CloudWatch metrics namespace,
- Return to the CloudWatch console. On the All metrics tab, under Custom Namespaces, choose DynamoDBStorageMetrics.
- Choose TableName, and select a table from the list to see the storage use. In this example, I ran this solution for few days to get metrics about my production tables.
The following screenshot shows storage metrics from the last week for the
And the following screenshot shows the current size of the
These metrics can help you to take corrective action, for example:
- You can see if your table has experienced sudden growth. If this is the case, you should check with your development team to understand whether the growth represents “the new normal,” or if recent code changes might have introduced issues inadvertently.
- If a table exceeds the acceptable storage size, you can configure an alarm on the associated CloudWatch metrics.
In this post, I demonstrated how to deploy a serverless application to monitor the storage of your DynamoDB tables. You can enhance this solution by, for example, sending Amazon SNS notifications from Lambda to your operations team when a table exceeds a specified size.
If you have questions about the implementation of this solution, submit them in the Comments section below.
About the Author
Masudur Rahaman Sayem is a solutions architect at Amazon Web Services. He works with AWS customers to provide guidance and technical assistance about database projects, helping them improve the value of their solutions when using AWS.