AWS Database Blog

Introducing Amazon S3 and Microsoft Azure SQL Database Connectors in AWS Database Migration Service

We are excited to announce the addition of two new database connectors in AWS Database Migration Service (AWS DMS)—Amazon S3 as a source and Microsoft Azure SQL Database as a source. You can now migrate data from these two new sources to all AWS DMS supported targets.

Amazon S3 as a source
You can now replicate data from an Amazon S3 bucket to our supported targets. These targets include relational databases (such as Oracle and Amazon Aurora), a data warehouse (Amazon Redshift), a NoSQL database (Amazon DynamoDB), or an Amazon S3 bucket. After performing a full load of the data files in the Amazon S3 bucket, AWS DMS can optionally replicate data changes to the target endpoint (see more information later in this post).

How to add an Amazon S3 source to AWS DMS
To add an Amazon S3 source to AWS DMS, you must provide access to an Amazon S3 bucket that contains one or more data files and a JSON document that describes the format of the data in those files.

The source data files must be in comma-separated-values (CSV) format and must be present in the Amazon S3 bucket before the full load starts.

  1. Sign in to the AWS Management Console, and choose Database Migration Service.
  2. In the navigation pane, choose Endpoints, and then choose Create endpoint.
  3. Choose the following:
    1. Endpoint type: Source
    2. Source engine: s3
  4. For Service Access Role ARN, provide an AWS account that has read access to the Amazon S3 bucket that you’re using as a source. For more details, see Prerequisites When Using S3 as a Source for AWS DMS.
  5. Add a bucket name and (optional) a bucket folder to define the path from AWS DMS to find the CSV files.
  6. For the External Table Definition, define or upload a JSON document that explains the structure of the source tables and columns in the source files.For example:1, Warriors, Celtics, Lakers2, Patriots, Seahawks, CardinalsThis translates into the following: 
    {
  {
    "Id":1,
    "Data1":Warriors,
    "Data2":Celtics,
    "Data3":Lakers,
  },
  {
    "Id":2,
    "Data1":Patriots,
    "Data2":Seahawks,
    "Data3":Cardinals,
  }
}

Using CDC with an Amazon S3 source
After performing a full load using the data files in your S3 bucket, AWS DMS can optionally replicate data changes to the target endpoint. To do this, you upload change data capture (CDC) files to your Amazon S3 bucket. AWS DMS reads these CDC files when you upload them and applies the changes at the target endpoint.

Use the CDC Path parameter to indicate the S3 path where the CDC files can be found. The files themselves must be named as follows:

<cdcpath>/CDC00001.csv

For more information about how to define an Amazon S3 bucket as an AWS DMS source, see Using Amazon S3 as a Source.

Azure SQL Database as a source
As mentioned earlier, you can now replicate data from a Microsoft Azure SQL Database to our supported AWS DMS targets. (For a full list, see Targets for AWS Database Migration Service.) With this release, we support only the full load option for Azure SQL Database.

How to add an Azure SQL Database as a source to AWS DMS
Follow these steps to add an Azure SQL Database source to AWS DMS:

  1. Sign in to the AWS Management Console, and choose Database Migration Service.
  2. In the navigation pane, choose Endpoints, and then choose Create Endpoint.
  3. Choose the following:
    1. Endpoint type: Source
    2. Source engine: azuredb (Azure SQL Database)
  4. The remaining parameters are the same as connecting to a Microsoft SQL Server database.

For more information about defining Azure SQL Database as an AWS DMS source, see Using Microsoft Azure SQL Database as a Source.

Learn more
We recently crossed 35,000 migrations and celebrated by writing a blog post that summarizes our latest releases. To read more, see Database Migration Gaining Momentum.

For more information about AWS DMS and the AWS Schema Conversion Tool, see the AWS website.

About the Author

Eran Schitzer is a product manager for Database Migration Services at Amazon Web Services.