How do I troubleshoot high source latency on an AWS DMS task?
Last updated: 2019-10-01
How do I troubleshoot high source latency on an AWS Database Migration Service (AWS DMS) task?
You can monitor your AWS DMS task using Amazon CloudWatch metrics. During a migration, you might see source latency during the ongoing replication phase—change data capture (CDC)—of an AWS DMS task. You can use the CloudWatch service metric for CDCLatencySource to monitor the source latency for an AWS DMS task. If you see source latency on an AWS DMS task, it can be caused by one or more of the following:
- The source database has limited resources.
- The AWS DMS replication instance has limited resources.
- The network speed between the source database and the AWS DMS replication instance is slow.
- AWS DMS reads new changes from the transaction logs of the source database during ongoing replication.
- AWS DMS task settings are inadequate or large objects (LOBs) are being migrated.
- The Oracle source database used for the AWS DMS task is using LogMiner for ongoing replication.
The source database has limited resources. It's a best practice to use native monitoring for your source DB engine to be sure that your database isn't experiencing a performance bottleneck, for example, CPU or memory contention or IO saturation.
The AWS DMS replication instance has limited resources. Monitor the replication instance metrics, such as CPUUtilization, FreeStorageSpace, and FreeableMemory. Confirm that the replication instance has enough resources to manage your task.
The network speed between the source database and the AWS DMS replication instance is slow. By design, a single AWS DMS task can't use the full network bandwidth. If you have a busy production database that has a high rate of changes, then you might need to increase the network bandwidth. For example, use AWS Direct Connect connections.
AWS DMS reads new changes from the transaction logs of the source database during ongoing replication. Depending on your source DB engine, the source transaction log can also have uncommitted data. During ongoing replication, AWS DMS reads incoming changes from the transaction logs, but AWS DMS forwards only committed changes to the target. Eventually, this can result in source latency.
However, when the source database writes a large dataset and executes fewer commits, AWS DMS continues reading from the transaction log. But AWS DMS doesn't apply changes the target until the entire transaction is committed, which can also cause source latency. Because the source latency increases, target latency also increases.
AWS DMS task settings are inadequate or large objects (LOBs) are being migrated. AWS DMS migrates LOB data for ongoing replication in two phases. First, AWS DMS creates a new row in the target table with all columns except those that have LOBs. Then, AWS DMS updates the rows that have LOBs. If you have a source database that frequently updates tables that have LOB columns, then you might see source latency. For more information, see Migrating Large Binary Objects (LOBs).
The Oracle source database used for the AWS DMS task is using LogMiner for ongoing replication. If you have a busy source database that generates a large number of redo logs or if the source database is using Oracle Automatic Storage Management (ASM), you might need to use the binary reader method for ongoing replication. For more information, see Using Oracle LogMiner or Oracle Binary Reader for Change Data Capture (CDC).