Automate copying AWS DMS tasks across AWS accounts

We are in the midst of digital transformation, and cloud adoption plays a big role in the journey to digital transformation. AWS Database Migration Service (AWS DMS) is a cloud service that makes it easy to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud or between combinations of cloud and on-premises setups. However, when working with different environments (like dev, staging, UAT, and production), it becomes time consuming to manage AWS DMS task promotion across environments especially if you prefer to modify the tasks for different testing and production environments. Manually copying AWS DMS tasks between AWS accounts as you promote the solution across each environment is a tedious and time-consuming task. For this, you need an automated solution to reduce manual repetitive tasks and errors.

In this post, we present a command line tool named the DMS Task Copy tool, which helps you copy AWS DMS task resources from a source AWS account that you want to promote to the next higher environment during your testing and production deployment phase.

The DMS Task Copy tool has two modes:

Export – Export table mappings and task settings from source AWS account into JSON files
Import – Create the AWS DMS task in target AWS account with provided table mapping and task setting JSON files

Furthermore, you can export the table mapping and task setting JSON files and use them to create an AWS CloudFormation stack for migration tasks.

Solution overview

The following diagram shows the high-level architecture and flow for the tool.

The DMS Task Copy tool authenticates with the AWS Identity and Access Management (IAM) or corporate Active Directory service and obtains a temporary authentication token via AWS Security Token Service (AWS STS). The tool then uses the token to access the AWS DMS tasks, replication instances, and endpoints on the source account.

The tool is designed to run in two modes: import and export modes. You should always run the tool in export mode first. During export mode, the tool connects to the source account and uses describe calls on the user tagged AWS DMS tasks and stores their table mappings and replication task settings as a JSON string into separate files on the local file system.

The table mapping files have a prefix of tm and are of the format tm-<dms-task-name>.json, where <dms-task-name> is the name of the AWS DMS task from the source account. Similarly, the replication instance settings files have a prefix of rts and are of the format rts-<dms-replication-instance-name>.json.

The tool places all the AWS DMS task files into a subfolder that gets created under the same directory where the Python scripts are located. The subfolder name is part of the configuration file and can be modified as required. The default name of the subfolder is dmstaskdata.

After the export step, you can edit the table mappings or replication task settings JSON files as required and then invoke the tool in import mode. The tool reads the JSON files from the subdirectory provided in the configuration file to recreate the AWS DMS tasks on the target account. This allows you to modify the task settings before they’re applied on the target account. You can also run the tool in import mode without any modifications to the table mappings or replication task settings. In that case, the tool reuses the exact settings from the source account to create the tasks on the target account.

The workflow to copy AWS DMS tasks across accounts includes the following steps:

Clone the GitHub repository for the DMS Task Copy tool.
Modify the config file config.json to provide the source and target account details and tags. See the complete details in configuration section later in this post.
Set a tag for the AWS DMS tasks, replication instances, and endpoints in the source account.
The Python script uses an AWS Identity and Access Management (IAM) role to connect to AWS DMS and retrieve the replication instance, endpoint and task details.
Set the AWS CLI to use the profile for the AWS source account. This will allow the tool to connect with the source account for extracting DMS task details.
Start the Python script in export mode to get the source account AWS DMS task details based on the defined tags. The tool copies the AWS DMS task definitions into the local directory.
After the AWS DMS tasks are exported, you can optionally edit the exported JSON files as required for the target environment like source and target endpoint details.
Set the AWS CLI to use the profile for the AWS target account. This will allow the tool to connect with the target account for creating DMS tasks.
Start the Python script in import mode to upload the AWS DMS tasks to copy to the target environment.
Validate the AWS DMS tasks copied to the target environment using the AWS Management Console and start the tasks as necessary.

For a detailed description of the above steps please refer to the DMSTaskCopy README.docx available on the GitHub repo.

Prerequisites

To implement this solution, you must have the following prerequisites:

Python 3.4 version or higher installed in your system.
AWS SDK for Python (Boto3) installed.
At least two different accounts with IAM roles setup with sufficient privileges and access to AWS DMS resources.
IAM user in the AWS source and target accounts with ‘AssumeRole’ permission on the above IAM roles.
Resource tagging at the source AWS account: AWS DMS source endpoint, AWS DMS target endpoint, AWS DMS replication instance, AWS DMS tasks.
A replication subnet group on the target account and provided in the configuration file for the rep_subnet_grp_identifier option under rep_instances.

Prepare the JSON configuration file

You must add tags to the AWS DMS tasks, endpoints, and replication instances in the source account and define it in the configuration file of the tool for the rep_task_tag_filters option, as described later in this section. The tool then reads the tasks and other related resources using the tag. The subdirectory name from the configuration file is used for placing the table mappings and replication task settings files underneath it.

The main advantage of using the DMS Task Copy tool is its flexibility in being able to modify attributes related to AWS DMS endpoints, replication instances, and task settings.

The configuration file follows JSON format. The following tables highlight a few of the important options.

Source Account Settings
Parameter	Value	Optional	Description
`src_account_id`	AWS account ID	No	Source account ID from which AWS DMS tasks must be copied
`src_account_role`	AWS account role	No	Predefined IAM user role such as `admin` or `devops`
`rep_task_tag_filters`	Tag	No	AWS tag defined on AWS resources to be copied in the source account

Target Account Settings
Parameter	Value	Optional	Description
`dest_account_id`	AWS account ID	No	The destination account ID
`dest_account_role`	AWS account role	No	The destination account role
`dest_environment`	Alphanumeric	No	Any unique identifier for the target environment to which AWS DMS tasks are copied or promoted such as `qa`, `uat`, or `prod`
`dms_task_import_export_subdir`	Folder name	No	A subfolder that the script creates to download the AWS DMS task table mapping and task setting json files

Generic Settings
Parameter	Value	Optional	Description
`ad_username`	AD login	No	The Active Directory user name. The password must be defined as an environment variable.
`dms_task_import_export_subdir`	Folder name	No	A subfolder that the script creates to download the AWS DMS task table mapping and setting json files.
`promt_start_dms_tasks`	True/False	No	Setting to `true` causes the script to prompt and wait for user input and confirmation before starting each AWS DMS task.
`auto_start_dms_tasks_on_creation`	True/False	No	Setting to `true` automatically starts all AWS DMS tasks after creation without waiting for user confirmation.
`ad_authentication`	True/False	No	Toggles between AD-based authentication or IAM. The default is `False`. If set to `True`, the `identity_service_url` must be set to point to the AD-based authentication REST service endpoint.
`sts_src_role_arn`	ARN	Yes	IAM role ARN from the source account. Required only when `ad_authentication` is set to False.
`sts_tgt_role_arn`	ARN	Yes	IAM role ARN from the target account. Required only when `ad_authentication` is set to False.

For a detailed description of the entire list of configuration options available, refer to DMSTaskCopy_README.docx available on the GitHub repo.

Run the tool

You must place the configuration file in the same directory as where the tool is run. A sample configuration file is available along with the source code in the GitHub repo.

Use the following command to export the AWS DMS tasks from the source account based on the tags defined for the tasks and endpoints into JSON files to your local directory:

python DMSTasksCopy.py --mode export

The following screenshot shows the exported table mapping and replication task setting files.

Import the AWS DMS tasks into the target environment with the following code:

python DMSTasksCopy.py --mode import

Import mode has two sub-options:

It can clean up existing tasks that it created during previous runs.

The script prompts you as shown below to check if a cleanup of resources is required:

Do you want to perform cleanup of resources before starting? (y/n) : y

It’s assumed that when the script is run for a second time; it cleans up the resources created by it in the target account.

Finally, you’re prompted to exit after cleaning. You may choose Y to exit the process after cleanup or N to continue importing AWS DMS tasks into target AWS account.

It can also start a replication task on creation.

The script (in import mode) also prompts if you want to start the replication task:

Do you want to start the replication task - fullload, (y/n)?y

If you choose N, it doesn’t start the replication task, which remains in a ready state.

The endpoints created by the tool in the target account are not tested for connections unless the prompt_start_dms_tasks option in the configuration file is set to true. In such cases, the tool also prompts you to start the task as shown above (for each copied task). Based on your choice (Y or N), it establishes the endpoint connections (if not already attempted) and starts the task.

Tool output

The output of the tool when run in different modes is as follows:

Export – You can see the AWS DMS task table mapping and replication task setting JSON files in your local directory. The files are placed under a subfolder whose name is configured through the config.json file.
Import – You can see the AWS DMS tasks, endpoints, and replication instances created in the target account, as shown in the following screenshots.

The following screenshot shows the tool creating the AWS DMS endpoints.

The following screenshot shows the tool creating the AWS DMS replication instance.

The following screenshot shows the tool creating the AWS DMS tasks in the target account.

The tool also generates a log file in the same directory as where its run. The output from the tool is also shown on the terminal window.

Summary

It can be time-consuming task to recreate AWS DMS tasks along with their table mappings, task settings, replication instances, and corresponding endpoints in the target AWS account during deployment. This tool can help reduce the manual work by offering a one-time setup through a configuration file.

Try the tool for your next database migration project during deployments across your different environments. We look forward to your comments and suggestions for improving it. Please explore the tool’s GitHub repository for a quick guide on getting started and running the tool.

About the Authors

Wasim Shaikh is a Database Specialist Solutions Architect with AWS. He works with customers to provide guidance and technical assistance about various database and analytical projects, helping them improving the value of their solutions when using AWS.

Rambabu Karnena is a Lead Consultant with AWS ProServe, GCC India. He works with customers in their journey to the cloud with a focus on complex database migration programs and Helping them improving the value of their solutions when using AWS.

Madan Revoor is a Lead Consultant with AWS ProServe, GCC India. He helps customers migrate their on-premise applications to the AWS cloud. He is passionate about DevOps and works with customers on automating their application deployments to EKS clusters.

AWS Database Blog