How to Scale AWS Database Migration Service (DMS) replication instances

AWS Database Migration Service (DMS) helps you migrate databases to AWS quickly and securely. The AWS DMS migration process encompasses setting up a replication instance, source and target endpoints, and replication tasks.

Your replication instance uses resources like CPU, memory, storage, and I/O, which may get constrained depending on the size of your instance and the kind of workload.

In this post, I show how you can automatically scale an AWS DMS replication instance to handle a higher load (scale up) when required and save money (scale down) when the load is low.

The use case

When setting up an AWS DMS replication instance, you likely analyze the following:

The number of tables in the database
The volume of data in those tables
Number of concurrent replication tasks
Traffic to the source database

In order to have the AWS DMS replication instance right-sized, you must be able to predict the right resource utilization (CPU).

Dynamic sizing solution overview

Here is the diagram of the architecture.

The AWS DMS best practices whitepaper outlines a number of strategies to set up a right-sized AWS DMS replication instance. In this post, I show how to achieve great flexibility in sizing the AWS DMS replication instance.

I use Amazon CloudWatch to monitor an AWS DMS replication instance for CPU utilization. After a CloudWatch alarm threshold is reached, it triggers Amazon Simple Notification Service (Amazon SNS) notification, which is subscribed to by an AWS Lambda function that will modify the replication instance. The function also indicates whether the tasks running on the new replication instance started successfully or not.

AWS DMS is a region-based service. If you want to use multiple regions, you will need to set up your alarms and resources in each AWS Region separately.

Getting started and prerequisites

To get started with this solution, you’ll need a AWS account. Follow the CloudFormation user guide to sign up and learn about pricing.

To log in to the AWS CloudFormation console, follow the instructions in the CloudFormation user guide. Use a CloudFormation template by choosing from the CloudFormation console to set up the environment or just follow the blog to dive deeper and understand how it works.

Some of the resources deployed in this blog post, including those deployed using the provided CloudFormation template, will incur costs as long as they are in use. Be sure to remove the resources and clean-up your work when you’re finished in order to avoid unnecessary cost.

Step 1: Create Amazon Simple Notification Service topic

In this section, create an SNS topic to which the Lambda function that you will create in step 5 publishes notification. The notification indicates updates to the replication instance. You can see this in item 7 in the architecture diagram.

From the AWS Management Console, under Services, choose SNS to access the SNS console then choose Topics on the left side of the page and then choose Create New Topic.
Enter Topic Name and Display name and choose Create topic For Example: Topic name: Replication_Instance_Update

To receive instance resize notifications, subscribe to this topic with an email address.

Step 2: Create another Amazon Simple Notification Service (SNS) topic

In this section, I demonstrate creating an SNS topic to which a Lambda function subscribes.

From the AWS Management Console, under Services, choose SNS to access the SNS console then choose Topics on the left side of the page and then choose Create New Topic.
Enter Topic Name and Display name and choose Create topic For Example:Topic name: dmsautoscaler
Display name: dmsauto

Step 3: Create CloudWatch alarm

The following steps demonstrate how to set up CloudWatch alarm to monitor AWS DMS replication instance CPU utilization.

Sign into the AWS Management Console and under Services, choose CloudWatch to access the CloudWatch console then choose the Alarms link on the left side of the page and then choose the Create Alarm button.
Select DMS under the Browser Metrics, CPUUtilization under Search metrics and choose Per-Replication Metrics and choose Next.The name of my replication instance is test.
You can now set up your CloudWatch alarm for this instance. In this example, shown in the figure below, I configured the alarm with the following settings and chose Create Alarm:Name: dms_cpu_highDescription: High CPU usage by DMS instanceWhenever CPUUtilization is: >= 80For: 5 data pointsPeriod = 1 MinuteStatistic = Average
Similarly, create another alarm for low CPUUtilization with the following settings and choose Create Alarm:Name: dms_cpu_lowDescription: low CPU usage by DMS instanceWhenever CPUUtilization is: <= 40For: 5 data pointsPeriod = 1 MinuteStatistic = Average

Step 4: Create an IAM role for the Lambda function

In this section, I demonstrate how to set up permissions needed for the Lambda function. I set up a custom AWS Identity and Access Management (IAM) policy and role to support these actions and assign them to the Lambda function that we’ll create in the next section. From the AWS Management Console, under Services, choose IAM to access the IAM console

Under Policy, create a policy with the permissions from the repo https://github.com/aws-samples/aws-dms-autoscaling/lambda_policy.json.It provides permissions for the Lambda function to:
1. Modify DMS replication instances
2. Log to CloudWatch Logs
3. Create and use CloudWatch event rules
4. Publish to SNS topic
Enter Name and description for your policy under Review policy. For ex: I have given it a name of dmsautoscaler.
Create a role for your Lambda function. Choose Roles on the left of the IAM console and then choose Create role
1. Select Lambda from the list of services and choose Next: Permissions
2. Select the box next to the policy you created previously i.e. dmsautoscaler, and choose Next: Review
3. Name your role and give it a description, then choose Create Role. In this example, I’ve given it a name of dmsautoscalerRole

Step 5: Create a Lambda function

In this section, I create a new Lambda function that will be subscribing to SNS topic dmsautoscalerRole that I created earlier. When the replication instance CPU utilization reaches the threshold that is set for CloudWatch alarm, it triggers an SNS notification, which is subscribed to by this Lambda function. The function scales the AWS DMS replication instance. Automatic scaling configuration is managed through a JSON file – instance_types.json that is stored in the Amazon owned S3 bucket – https://s3.amazonaws.com/aws-database-

blog/artifacts/auto_scale_DMS_replication_instance/instance_types.json . However, if you would like to control how to scale the replication instance, then you can download the file from the Amazon S3 bucket and modify it as shown in this sample:

"dms.t2.micro": {
        "cpu_high": "dms.t2.small", 
        "cpu_low": "no_action"
    },
    "dms.t2.small": {
        "cpu_high": "dms.t2.medium",
        "cpu_low": "dms.t2.micro"
    },

Feel free to change instance types for any of the below and store it in your own Amazon S3 bucket. Point the Lambda function to it through environment variables.

If the next replication instance type is available, the Lambda function modifies the existing DMS replication instance with it. In addition, it creates an Amazon CloudWatch scheduled event that invokes the Lambda function every minute to poll AWS DMS replication instance. This checks the status of the replication instance and its tasks.

From the AWS Management Console, under Services, choose Lambda to access the Lambda console.
From the dashboard, choose Create Function or, if you were taken directly to the Functions page, choose the “Create Function” button on the upper-right corner.
On the Create function page:
1. Choose Author from scratch
2. Give your function a name. I chose AutoscaleDMSReplicationInstance
3. The Lambda function I’ve written is Python 3.7 compatible, so choose Python 3.7 in the Runtime dropdown
4. Under Role select Choose an existing role and select the role created in the previous section – dmsautoscalerRole, then choose Create function
5. Next add SNS trigger by choosing SNS in the Add triggers section. An SNS box should appear connected to the left side of the Lambda function, and have a note stating Configuration required. I add the SNS topic that you created in the step 2 – dmsautoscaler.
Scroll up to the Designer section and choose the name of your Lambda function AutoscaleDMSReplicationInstance. Paste the Python code from https://github.com/aws-samples/aws-dms-autoscaling/dms.py under the Function Code section that appears.

Scroll down to the Environment variables Create following environment variables:

	Key	Default Value	Description
1	BUCKET_NAME	aws-database-blog	The Lambda function picks up the configuration file from here.
2	KEY_NAME	artifacts/auto_scale_DMS_ replication_instance/insta nce_types.json	This is the configuration file name.
3	MODIFICATION_TIMEOUT	1200	Total timeout value after which the Lambda function notifies the user that replication instance modification timed out.
4	TOPIC_ARN	arn:aws:sns:us-west- 2:<<AccountId>>:Replicati on_Instance_Update	SNS ARN for the Replication_Instance_Update topic.

Under Basic Settings, set the Timeout to 30 sec.
Leave everything else as-is, and choose Save at the top.

Conclusion

In this post, I demonstrated how you can use CloudWatch alarms together with Lambda functions to monitor the utilization of AWS DMS replication instances. You can see how easy it is to automatically scale the instances to an appropriate size.

If you have any questions or comments about this post, please use the comments section on this page.

About the Authors

Ballu Singh is a principal solutions architect at AWS. He lives in the San Francisco Bay area and helps customers architect and optimize applications on AWS. In his spare time, he enjoys reading and spending time with his family.

Prabhat Sharma is a solutions architect at AWS. He is based out of San Francisco and helps customers in San Francisco and Silicon Valley, design and build large scale applications on AWS. He has expertise in the area of AWS, containers and machine learning. He loves spending time with his son and wife.

Arun Thiagarajan is a database engineer with the AWS DMS and AWS SCT team at Amazon Web Services. He works on challenges related to database migrations and works closely with customers to help them realize the true potential of AWS DMS. He has helped migrate hundreds of databases to the AWS Cloud by using AWS DMS and the AWS SCT.