Introducing log management in AWS Database Migration Service

AWS Database Migration Service (AWS DMS) helps you migrate databases to AWS quickly and securely. You can migrate your data to and from most widely used commercial and open-source databases, such as Oracle, Microsoft SQL Server, and PostgreSQL. The service supports homogeneous migrations such as Oracle to Oracle, and also heterogeneous migrations between different database platforms, such as Oracle to PostgreSQL or MySQL to Oracle. In addition, it gives you a way to check aspects of your migration by using a premigration checklist and data validation.

We are excited to announce a new feature that helps manage migration task logs in AWS DMS.

Why you might want to manage logs
Every AWS DMS migration task has a log that records all the events, checkpoints, and task modifications performed while running the migration. The task logs are a critical component of any migration, and understanding and managing these logs is vital. They contain invaluable information that can help you gain an understanding about what is going on under the hood and determine the root cause almost immediately in many cases.

By default, the DMS task logging level is minimal. However, sometime this minimal level of logging might not help determine what caused a migration to fail or error. For troubleshooting, you can make logging more granular to catch the issue causing failures or errors.

The caveat about changing the default logging level to detailed is that this causes storage to get consumed rapidly. This happens because at the detailed logging level, DMS writes each and every event to the log files, which it doesn’t do at the default logging level.

When generating logs for migration tasks, AWS DMS creates the logs on the replication instance. These logs reside on the underlying disk for the replication instance. Given that AWS DMS is a fully managed service and doesn’t allow access to the replication instance, you can see these logs only by using Amazon CloudWatch. For each replication instance, AWS DMS deletes logs that are older than seven days.

If not correctly managed, these logs can consume a lot of storage resources and can fill up the underlying disk, which eventually interrupts the running migration tasks. Up until now, to get the tasks running normally again, you had two options:

• Allocate more storage. If a task is stuck due to storage being full, you allocate more space by modifying the replication instance storage.
• Delete logs. If you want to get the logs purged and don’t want to modify the storage, then you reach out to AWS Support. You then wait until the support team can investigate the issue and resolve it.

How to manage migration logs
To help make AWS DMS completely managed, AWS has introduced a feature to manage migration logs on the replication instance. You can also use this feature to find the amount of storage consumed by logs for each task for a particular replication instance. In addition, this feature lets you purge log files at your convenience.

Managing logs by using the AWS DMS console
In the AWS DMS console, you can manage logs on the replication instance page. The new Log management tab lists all the tasks and the total memory occupied by logs for each task. To allow log purging, each task has a check box and a Delete Logs button at the page bottom.
Managing logs by using the AWS DMS CLI

If you want to use the AWS CLI to manage logs, use the following command:

awsdms describe-replication-instance-task-logs --replication-instance-arn $REP_INSTANCE_ARN

The preceding command results in the following output:

{
    "ReplicationInstanceTaskLogs": [
        {
            "TaskName": "J3BYRX6OIZDPUO4NW6D5WR3OFU",
            "ReplicationInstanceLogSize": 12033
        },
        {
            "TaskName": "NRVPM7SD5CCIFNO6BLMYB4AMRA",
            "ReplicationInstanceLogSize": 10068
        },
        {
            "TaskName": "QLPXAVV4AB5BL5HAGDMDB7OEZM",
            "ReplicationInstanceLogSize": 5034
        }
    ],
    "ReplicationInstanceArn": "arn:aws:dms:us-east-1:123456789101112:rep:FKQACMLFACABIMQE55GEFIM3KU"
}

As evident from the preceding output, the replication instance contains three tasks. The output also shows the storage space occupied by each task, in megabytes.

Summary
By using log management in AWS DMS, you can gain a high level of confidence in your migrations. You can also use log management determine how much of your resources are consumed by each task and to control that resource consumption. In addition, you can use log management to get metrics, such as the number of tasks on each replication instance and corresponding storage occupancy, which help provide fine-grained control over your migration.

Good luck, and happy migrating!

About the Author

Abhinav Singh is a database engineer in Database Migration Service at Amazon Web Services. He works with our customers to provide guidance and technical assistance on database migration projects, helping them improving the value of their solutions when using AWS.