AWS Database Blog

VPC endpoint considerations for upgrading or creating AWS DMS version 3.4.7 or higher

AWS Database Migration Service (AWS DMS) version 3.4.7 and above includes support for Amazon Virtual Private Cloud (Amazon VPC) VPC endpoints and makes it easier to maintain end-to-end network configuration within the same AWS region for replication tasks. To learn more about VPC endpoints, refer to Connect your VPC to services using AWS PrivateLink.

In this post, we demonstrate an AWS DMS task failure caused by a network issue that you may encounter while upgrading your AWS DMS replication instance to version 3.4.7 or higher. We also show how to mitigate this issue by configuring a VPC endpoint and upgrading the AWS DMS replication instance for seamless replication.

Solution overview

VPC endpoints for all source and target endpoints ensures that all your traffic remains within your VPC and under your control. It reduces replication interruptions and improves the quality of the data transfer. With VPC endpoints, your AWS DMS replication instance doesn’t require a public IP address to communicate with AWS services such as Amazon Simple Storage Service (Amazon S3), Amazon Kinesis, AWS Secrets Manager, Amazon DynamoDB, Amazon Redshift, and Amazon OpenSearch Service.

The following diagram illustrates the solution architecture.

Prerequisites

To follow along with this post, you should have the following prerequisites:

Migration with AWS DMS 3.4.6

We created an AWS DMS task for a sample (DMS_SAMPLE) schema and performed a full load and change data capture (CDC) to migrate data from the source database to the Amazon S3 target.

From the AWS DMS task logs (see the following screenshot), we can see that the task for table TESTING_4 is running fine.

Next, we upgrade the AWS DMS replication instance to 3.4.7.

Upgrade the AWS DMS replication instance to 3.4.7

You can upgrade the replication instance by modifying the replication instance and specifying the replication engine version as 3.4.7.

After the upgrade is complete and the task is resumed, you see the following error messages in the AWS DMS task logs.

The reason for the failure is that, for private AWS DMS Replication instances prior to AWS DMS version 3.4.7, the replication network traffic for the S3 endpoints were routed through the Amazon backbone network. Starting with AWS DMS version 3.4.7 and above, you need VPC endpoints or an internet gateway route in the route tables used by AWS DMS replication instance to access the Amazon S3 target. VPC endpoints powered by AWS PrivateLink allow you to connect directly and privately to AWS services like Amazon S3 from within your VPC.

In the next step, we create a VPC endpoint.

Create a VPC endpoint

To create a VPC endpoint, complete the following steps:

  1. On the Amazon VPC console, choose the same Region as your AWS DMS replication instance.
  2. In the navigation pane, choose Endpoints.
  3. Choose Create endpoint.
  4. For Service category, select AWS services.
  5. Under Services, filter for Amazon S3.
  6. Select the Amazon S3 internet gateway.

This creates a gateway endpoint that sends traffic to Amazon S3 using a private IP addresses. You route traffic from your VPC to the gateway endpoint using route tables which we select below.

  1. For VPC, choose the same VPC as the AWS DMS replication instance.
  2. For Route tables, select the applicable routes.
  3. Under Policy, select Full access.
  4. After the endpoint is created, verify the status shows as Available.

Test the Amazon S3 connection endpoint

Test the Amazon S3 target endpoint with the newly upgraded 3.4.7 replication instance and check if it’s successful.

Now you can resume/restart the AWS DMS task and verify the task is running fine. You can also, check the AWS DMS task CloudWatch logs to confirm that the task is running as expected.

Clean up

Remove the resources that were created when testing the upgrade to avoid any ongoing charges.

Recommendations when upgrading an AWS DMS Replication Instance from 3.4.6 to 3.4.7

  1. Create a VPC Endpoint
  2. Stop the AWS DMS task/s that are currently running on the AWS DMS replication instance you wish to upgrade
  3. Upgrade the AWS DMS Replication Instance from 3.4.6 to 3.4.7
  4. Resume the AWS DMS task/s that were stopped

Summary

In this post, we provided step-by-step guidance on safely upgrading your AWS DMS replication instance to 3.4.7 and using VPC endpoints to communicate with Amazon S3 as a target endpoint.

Leave a comment if you have questions or suggestions.

About the authors

Sushant Deshmukh is a Database Consultant with AWS Professional Services. Sushant works with AWS customers and partners to build highly available, scalable and secured database architectures on AWS. Also, helping customers migrate and modernize their databases to AWS Cloud. Outside of work, he enjoys traveling, exploring new places, playing volleyball and spending time with his family and friends.

Aritra Biswas is a Cloud Support DBA with Amazon Web Services and Subject Matter Expert for AWS Database Migration Service, he has over a decade of experience in working with relational databases. At AWS, he works with Service Teams, Technical Account Managers, Solutions Architects, and assists customers migrate database workloads to AWS. Outside of work he enjoys playing racquetball and spending time with family and friends.

Arnab Saha is a Senior Database Specialist Solutions Architect at Amazon Web Services. Arnab specializes in Amazon RDS, Amazon Aurora , AWS DMS and Amazon Elastic Block Store. He provides guidance and technical assistance to customers thus enabling them to build scalable, highly available and secure solutions in AWS Cloud.