How can I troubleshoot high target latency on an AWS DMS task?

Short description

Use Amazon CloudWatch metrics to monitor your replication task's metrics. To identify replication latency in the ongoing replication phase, monitor CDCLatencySource and CDCLatencyTarget. The CDCLatencySource metric is the latency between the source and the replication instance. The CDCLatencyTarget metric is the latency between the replication instance and target. For more information, see Replication task metrics.

High CDCLatencySource means that the process to capture changes from the source is delayed. High CDCLatencyTarget means that the process to apply the change events to the target is delayed. If both CDCLatencySource and CDCLatencyTarget are high, then troubleshoot CDCLatencySource first because target latency is always the same or greater than the source latency. High CDCLatencyTarget is most likely because there's a delay to capture the change events from the source. If the CDCLatencySource isn't high but CDCLatencyTarget is, then the following issues can cause the latency:

• There are no primary keys or indexes in the target.
• There are resource bottlenecks in the target.
• There are resource bottlenecks in the replication instance.
• There's a network issue between the replication instance and the target.

To resolve these issues, see the Best practices and troubleshooting section in the following Resolution.

Resolution

No primary keys or indexes in the target

By default, AWS DMS uses data manipulation language (DML) statements to write changes to the target, such as INSERT, UPDATE, or DELETE. If the required indexes aren't in place, then changes, such as updates and deletes, can result in full table scans. Full table scans can cause performance issues on the target and result in target latency. Check your target database schemas, especially if you manually created the target schema. Use target database mechanisms to identify slow queries, such as the slow query log for MySQL, pg_stat_activity for Amazon Relational Database Service Amazon RDS) for PostgreSQL, or a query plan. If your target is Amazon Redshift, then also check the distribution style for your table. Because distribution styles take longer to insert or update data into tables, all distribution styles can cause target latency.

Resource bottlenecks in the target

If your target doesn't have sufficient resources, then the target can't accept changes at the rate that AWS DMS sends them. This can cause resource bottlenecks on the target and target latency. This also happens if other processes consume resources in the target. If AWS hosts the target, then check the resource statistics from the CloudWatch metrics.

Resource bottlenecks in the replication instance

Choose a replication instance with enough resources to handle your migration: CPU, memory, network, or iOPS.

Use CloudWatch metrics to monitor your replication instance resources.

Network issue between replication instance and target

Network bandwidth can also cause latency issues, especially when your target is an on-premises database, or if you use AWS DMS for cross-AWS Region replication.

Best practices and troubleshooting

If your target is Amazon RDS, then follow the best practices to improve the performance of an AWS DMS migration. Amazon RDS has an automated backup mechanism that starts within the backup window, and Amazon RDS backs up the moved data. If a snapshot of the target RDS DB instance is in the capture process, then AWS DMS can experience issues when the service applies changes to the target. As a result, the target latency increases until the snapshot capture completes. If your target is Amazon Elastic Compute Cloud (Amazon EC2) or an on-premises database, then check your target database's backup mechanism.

Some task settings cause changes to be written slowly to the target. If you run ongoing replication from a source where the rate of change is high, then use BatchApplyEnabled. For more information, see the BatchApplyEnabled section of Debugging your AWS DMS migrations: What to do when things go wrong? Part 3

To set BatchApplyEnabled to True, use the AWS Command Line Interface (AWS CLI) to run the modify-replication-task command:

aws dms modify-replication-task --replication-task-arn arn:aws:dms:ap-northeast-1:123456789012:task:ABCDEFGHIJKLMNOPQRSTUVWXYZ --replication-task-settings "{\"TargetMetadata\":{\"BatchApplyEnabled\":true}}"

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Related information

Change processing tuning settings