AWS Database Blog

Cross-account replication with Amazon DocumentDB (with MongoDB compatibility) and AWS DMS

Customers running their database workloads using Amazon DocumentDB (with MongoDB compatibility) might need to replicate data from one AWS account to another. By doing so, you can achieve faster development or QA environment refreshes, which provides a production-like environment to troubleshoot current production issues, and share data with partners.

Amazon DocumentDB provides support for copying snapshots between different AWS accounts. You can use this feature to copy data to other accounts, and the time taken to copy data varies depending on the size of the snapshot. Larger snapshots take more time because each copy transfers the full snapshot. An alternate approach can be to use AWS Database Migration Service (AWS DMS) or Amazon DocumentDB change streams to replicate data across accounts.

AWS DMS makes it easy to migrate relational and NoSQL databases to AWS. With AWS DMS, you can perform one-time migrations, and you can replicate ongoing changes to keep sources and targets in sync. AWS DMS supports Amazon DocumentDB as source and target endpoints to migrate data to and from Amazon DocumentDB. For more information, see Using Amazon DocumentDB as a target for AWS Database Migration Service.

In this post, I use Amazon DocumentDB as source and target database clusters, AWS DMS for data replication, and VPC peering with Amazon Virtual Private Cloud (Amazon VPC) to connect two AWS accounts. Later, I use a sample dataset to verify real-time data replication between the source and target.

Solution overview

The solution in this post addresses the following main concerns:

  • Real-time data replication between AWS accounts and AWS Regions
  • Creating a global solution for data sharing with partners, environment refreshes, and quickly provisioning a production-like staging area for troubleshooting

The following architecture diagram illustrates my solution.

The implementation of this solution consists of the following tasks using various AWS services:

  1. Create a cross-Region VPC peering connection using two different AWS accounts.
  2. Create an AWS DMS task to start a full load and change data capture (CDC) replication from the source Amazon DocumentDB database in one Region to the target Amazon DocumentDB database in a different Region.
  3. Run the replication task.
  4. Verify the changes are replicated from the source to the target.

Prerequisites

This post assumes the following AWS resources have already been provisioned:

Setup VPC peering

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4/IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. A VPC peering connection helps you facilitate the transfer of data between VPC resources, including Amazon Elastic Compute Cloud (Amazon EC2) instances and Amazon DocumentDB databases that run in different Regions. It’s important to note that you can’t create a VPC peering connection between VPCs that have matching or overlapping IPv4 or IPv6 CIDR blocks.

In this step, I create a VPC peering connection between two VPCs in different Regions using two different accounts. We start by accessing the AWS Management Console in Account A.

  1. From the Amazon VPC console, choose Peering Connections.
  2. Choose Create Peering Connection.
  3. Name the peering connection docdb-vpc-peering.
  4. For Select a local VPC to peer with, choose VPC (Requester)*.
  5. For Account, choose Another account and enter the account ID (target Account B).
  6. For Region, choose Another Region.
  7. Choose VPC ID (Accepter)*.
  8. Leave the Tags section at the default settings.
  9. Choose Create peering connection.
  10. Choose OK.
  11. On the AWS Identity and Access Management (IAM) console, under Virtual Private Cloud VPC), choose Peering Connections.
  12. Choose the new peering connection, which is in Pending Acceptance state.
  13. On the Actions menu, choose Accept Request.
  14. Choose Yes, Accept.
  15. Choose Close to complete the setup.

You can verify the status of this VPC peering connection is active in both accounts.

You must verify that you have Amazon DocumentDB clusters set up in the source and target accounts in order to complete the next steps. If they’re not set up, see Creating an Amazon DocumentDB Cluster.

Create a database migration task

An AWS DMS task is where all the work happens. Here you specify what collections and schemas to use for your migration. When creating a migration task, make sure that you create a source endpoint, a target endpoint, and a replication instance as mentioned in the prerequisites section.

Before you start the replication, make sure to enable the change streams on the source database in order for the migration task to run in CDC mode. Run the following code from the mongo shell on the source Amazon DocumentDB cluster:

db.adminCommand({modifyChangeStreams: 1,
    	       database: "",
            collection: "", 
            enable: true});

In this step, I create a migration task that starts the initial migration between the source and target data stores and captures ongoing changes to the source data store after completing the initial migration to the target data store. Let’s start by accessing the console in Account A.

  1. On the AWS DMS console, choose Replication instances and verify the replication instance.
  2. Choose Endpoints and verify the source and target endpoints.
  3. In the Task configuration section, choose Database migration.
  4. Choose Create task.
  5. For Task identifier, enter a name.
  6. For Descriptive Amazon Resource Name (ARN), provide a name.
  7. For Replication instance, choose an existing replication instance.
  8. For Source database endpoint, choose an existing endpoint.
  9. For Target database endpoint, choose an existing endpoint.
  10. For Migration type, choose Migrate existing data and replicate ongoing changes.
  11. In the Task settings section, for Editing mode, choose Wizard.
  12. For Target table preparation mode, choose Drop tables on target.
  13. For Stop task after full load completes, choose Don’t stop.
  14. Choose Include LOB columns in replication and choose Limited LOB Mode.
  15. For Maximum LOB size (KB), enter 32.
  16. Leave Enable validation
  17. Select Enable CloudWatch logs and leave the associated fields at their default.
  18. Under Advanced task settings, choose Create control table in target using schema.
  19. Use the schema migtest.
  20. For History timeslot, enter 5.
  21. Enable Apply exceptions/Replication status/Suspend tables/Replication history.
  22. In the Table mappings section, for Editing mode, choose Wizard.
  23. For Selection rules, choose Add new selection rule.
  24. For Schema, enter migtest.
  25. For Table name, enter %.
  26. For Action, choose Include.
  27. Leave Premigration assessment
  28. In the Migration task startup configuration section, for Start migration task, chose Manually later.
  29. Optionally, apply tags in the Tags
  30. Choose Create task.

At this stage, the migration task is being created.

Run the migration task

When the task is in Ready status, start the AWS DMS migration task.

  1. On the AWS DMS console, select your migration task.
  2. On the Actions menu, choose Restart/Resume.
  3. Choose the Table statistics tab to see the current activity that’s happening under this task.

Test the solution

At this stage, you’re ready to enter sample data in the source Amazon DocumentDB database (migtest). Following screen shot is showing the source DocumentDB before a sample data insert.

Similarly, in the mongo shell, connect to the target Amazon DocumentDB database (target-documentdb) and verify the latest data.

After you add the sample dataset on the source database under Account A, the AWS DMS replication task populates those changes to the target database under Account B, which is also in a different Region.

The following screenshot shows our source database insert statement –

The following screenshot shows the data replicated in our target database.

You can also verify the changes are being captured and replicated from the source to target database on the AWS DMS console, on the Quick view and compare page.

The following screenshot shows the Table statistics tab.

You can also track Amazon CloudWatch metrics of this migration task on the CloudWatch metrics tab.

On the Overview details tab, choose View CloudWatch logs to see detailed information of the changes being replicated.

Besides using the CloudWatch to monitor the data replication task, it’s always a good idea to keep an eye on DMS replication instance activity, use Amazon SNS (simple Notification Service) for DMS events notification when your workload is running. If you plan to use ongoing replication, we recommend using a Multi-AZ option for your DMS replication instance to provide high availability and failover support for the replication instance. To use AWS Database Migration Service most effectively, see DMS best practices on the most efficient way to migrate your data.

Summary

This post demonstrates how to replicate data across AWS accounts using Amazon DocumentDB clusters, AWS DMS, and VPC peering. You can also use this solution for data sharing and for database environment refreshes across accounts. You can further enhance this solution with AWS CloudFormation to automate spinning up the DMS resources and creating the migration task.

If you have any questions or comments about this post, please use the comments section. If you have any features requests for Amazon DocumentDB, email us at mailto:documentdb-feature-request@amazon.com

About the author

Pragnesh Patel is a Database Migration Consultant at AWS. He works with customers in their journey to the cloud with a focus on database migrations. In his spare time, Pragnesh enjoys traveling to new places with his wife and kids.