Migration Validation (Part 1) – Introducing Migration Assessment in AWS Database Migration Service

We are excited to announce a new feature that provides a pre-migration checklist in AWS Database Migration Service (AWS DMS). AWS DMS does a great job of helping you move your data between multiple supported sources and targets. However, migrations can be difficult, especially when you’re moving from one database engine to another (known as a heterogeneous migration). Multiple factors like resources (on the source, target, and replication instances), the network, and the nature of the data play a big part in determining the success of your database migration. Even if you’re doing a “simple” lift and shift, you can face many issues that might complicate the process.

There can also be limitations related to using AWS DMS. For example, AWS DMS might not support the data types in the tables involved in the migration. Or the prerequisites for the source or target database engine might not have been followed correctly. With the introduction of this feature, AWS DMS extends the managed service capability by providing you with a pre-migration task assessment. This capability helps you validate migration task settings before initiating your task.

This post discusses what aspects the migration assessment feature addresses and how you can use it before starting your migration.

Pre-migration checklist
The success of the data migration process is dependent on many variables—for example, the network connection between the components, the database permissions, and the source data that is being migrated. Some migration processes end with data that is only partially migrated. Some result in failures due to incorrect configurations on the source or target database engines or other problems that affect the success of the migration.

The task assessment report indicates any inaccuracies or problems that might occur during the migration. It helps identify problems early in the process so that you can tackle them accordingly.

The pre-migration task assessment is a list of tests. It performs validation checks of the different migration components and provides a list of warnings and errors. The objective of the checklist is to identify issues with the migration process before it starts running and to inform you of migration errors that are caused by data type issues between the source and the target.

Migration assessments
With the current release of this feature, the task assessment report includes information about unsupported and partially supported data types in AWS DMS. The assessment process scans all the source data types in the tables that are to be migrated. It then compares this list with the supported data types for this type of database and generates a report. This report indicates the data types that are not supported or that are partially supported for the present database migration.

To review the detailed task assessment report on the AWS DMS console, choose Open. The following image shows a sample report:

These results are categorized as follows:

• Not supported
  • Data types that AWS DMS does not support. Those columns of data won’t be migrated.
• Partially supported
  • Data types that AWS DMS supports, but when it tries to migrate them, the target data type mapping might not match the source. For example, when you move from PostgreSQL to Oracle, the TEXT data type is mapped as NCLOB on the target and not CLOB or BLOB.
  • Data types that AWS DMS can migrate, but with partial success. For example, when you move from PostgreSQL to PostgreSQL, JSONB migration works fine if the data in this column is fewer than 255 characters.

Additionally, the results are generated and stored as follows:

• The task assessment process creates an Amazon S3 bucket in the customer account with the name dms-<account-number>-<uniqueidentifier>.
• Each task assessment creates a folder with the task name and a subfolder with the date of the assessment.
• The data file includes a list of data structures in JSON format for each unsupported data type.
• The results file starts with a summary that includes a list of the unsupported data types and the column count of each one.

Enable task assessment on the console
As mentioned before, the assessment report is generated before the task is run. On the AWS DMS console, you enable the feature when you create the task.

When creating the task, under Post Creation, choose Task assessment. The following report is generated on the Assessment results tab:

After you review the report and make the necessary corrections, choose Start task to initiate the migration process.

Enable task assessment with the AWS CLI
If you use the AWS CLI, the task must already be created. If you use the AWS CLI for reviewing the pre-migration checklist, when you create the task, have it in the Start task mode. Then use the following command to start the assessment:

aws dms start-replication-task-assessment --replication-task-arn <task_arn>

Use the following command to review the generated assessment report:

aws dms describe-replication-task-assessment --replication-task-arn <task_arn>

The preceding command results in the following output:

{
    "ReplicationTaskAssessmentResults": [
        {
            "ReplicationTaskIdentifier": "mytask123", 
            "AssessmentStatus": "issues found", 
            "S3ObjectUrl": "https://dms... ", 
            "AssessmentResultsFile": "mytask123/2017-11-10-16-31", 
            "ReplicationTaskArn": "task_arn", 
            "AssessmentResults": "{\"summary\":{\"task-name\":\"mytask123\",\"not-supported\":{\"data-types\":[],\"column-count\":0},\"partially-supported\":{\"data-types\":[\"enum\"],\"column-count\":1}},\"types\":[{\"data-type\":\"enum\",\"support-level\":\"partially-supported\",\"schemas\":[{\"schema-name\":\"employees\",\"tables\":[{\"table-name\":\"employees\",\"columns\":[\"gender\"]}]}]}]}", 
            "ReplicationTaskLastAssessmentDate": 1510331499.0
        }
    ], 
    "BucketName": "dms-<customer_account_number>-uniqueid"
}

This command fetches the results from the S3 bucket where AWS DMS stores the assessment results and displays the output accordingly. If you run multiple task assessment reports, it will output all the reports.

Summary
Database migration projects can be difficult, but the benefits of migrating your database to the cloud are significantly greater than the challenges migrations can present. Using the migration assessment feature can help you predict issues with a migration before you start. It also helps keep you informed about what to expect from your migrations. The checklist generated by this feature helps you assess and make the necessary changes required by AWS DMS and helps put your migrations on the right track.

Thanks for reading! Have a successful database migration!

About the Author

Abhinav Singh is a database engineer in Database Migration Service at Amazon Web Services. He works with our customers to provide guidance and technical assistance on database migration projects, helping them improving the value of their solutions when using AWS.