How can I migrate my DynamoDB tables from one AWS account to another?
Last updated: 2020-09-24
How can I perform a cross-account DynamoDB table migration?
You can use AWS Data Pipeline or Amazon EMR to move DynamoDB tables to another AWS account. Data Pipeline is the fastest and easiest method, but provides fewer options for customization. Amazon EMR is a better choice for users with more technical expertise who want more control over the process.
To move a DynamoDB table to a different account using Data Pipeline, see How can I use Data Pipeline to back up a DynamoDB table to an S3 bucket that is in a different account?
Note: The destination account can't access the DynamoDB data in the Amazon Simple Storage Service (Amazon S3) bucket. To work with the data, restore it to a DynamoDB table.
When you use Amazon EMR to migrate DynamoDB tables, you have two options, depending on your use case:
- If you can afford downtime during the migration, then stop write operations to the source table to assure that the target table is in sync with the source table.
- If you can't afford downtime, then you must store all transactions that happen during the migration in a staging table. After the original table is migrated to the other AWS account, push the new transactions from the staging table to the target table.
Note: The time required to migrate tables with Amazon EMR can vary significantly depending on network performance, the DynamoDB table's provisioned throughput, the amount of data stored in the table, and so on.
To migrate a DynamoDB table using Amazon EMR:
- Launch EMR clusters in both the source and destination accounts. In the Software configuration section, be sure that you choose an option that includes Apache Hive.
Note: It's a security best practice to launch Amazon EMR clusters into private subnets. The private subnets must have an Amazon S3 VPC endpoint and a route to DynamoDB. For more information, see Private subnets. If the clusters need to access the internet, use a NAT gateway that resides in a public subnet. For more information, see VPC with public and private subnets (NAT).
- Be sure that the EMR_EC2_DefaultRole AWS Identity and Access Management (IAM) roles in both accounts have permission to write to the S3 bucket in the destination account. For more information, see Configure IAM service roles for Amazon EMR permissions to AWS services and resources.
- In the source account, connect to the master node using SSH.
- In the source account, use Hive commands to export the DynamoDB table data to the S3 bucket in the destination account.
- In the destination account, import the Amazon S3 data to the new DynamoDB table.
- If you're using a staging table to capture writes that happened during the migration, repeat steps 4 and 5 on the staging table.