Introducing Data Extractors in AWS Schema Conversion Tool Version 1.0.602

We are pleased to announce the recent release of version 1.0.602 of the AWS Schema Conversion Tool (AWS SCT). The new version includes a new key capability—data migration agents that extract data from Oracle, MS SQL Server, Teradata, IBM Netezza, Greenplum and HPE Vertica data warehouses, prepare, optimize and upload it to Amazon Redshift.

The new data migration agents are locally installed agents designed to extract data from data warehouses based on Oracle version 11, MS SQL Server version 2008 and later, Teradata version 14.10 and later, IBM Netezza version 7.0.3 and up, Greenplum version 4.3 and up and HPE Vertica version 7.2.2 and up. When data is extracted, the AWS Schema Conversion Tool optimizes it for Amazon Redshift and saves it in local files. AWS can automatically upload these files to Amazon Redshift or you can transfer these files to an Amazon Simple Storage Service (Amazon S3) bucket, where they can be uploaded to Amazon Redshift manually. For large datasets, you can use AWS Snowball to ship the data to AWS.

The migration agents are designed to extract data in parallel and independently. For example, with a large Oracle data warehouse SCT will automatically distribute work between all available migration agents. SCT will automatically manage all the available agents to extract the data from the different partitions and tables in the schema in the most optimized way for Amazon Redshift, consolidate the data, and save it in local files. The migration agents work completely independent from SCT, and you can replace them, if needed, without any work lost. This unique independent parallel execution capability not only accelerates the extraction of the data, but can also withstand failure of one of the agents.

The new migration agents are supported on RedHat and Debian Linux distros, and the SCT client can run on Windows, Mac, Fedora, or Ubuntu. For more information on installation, security settings, and configuration, see Installing Migration Agents in the AWS SCT User Guide.

When the migration agents are installed and configured, you manage them by using the AWS Schema Conversion Tool. The migration agents act as listeners. When they receive instructions from the AWS Schema Conversion Tool, they extract data from your data warehouse. You can find instructions on how to register the migration agents with your AWS Schema Conversion Tool project in Managing Migration Agents in the AWS SCT User Guide.

After your migration extraction tasks complete, your data is ready. The migration agents save your data as files that are compressed, but not encrypted. The migration agents can copy the data files to your Amazon S3 bucket over an Internet connection. Alternatively, you can copy the files to an Amazon S3 bucket using AWS Snowball. Copy the data from the output folder, under the working folder that you specified for your agents.

After uploading your data files to Amazon S3, the data is ready to be uploaded to Amazon Redshift. The extraction task can automatically upload the data to Amazon Redshift or you can use the Amazon Redshift COPY command to load your data to Amazon Redshift. For more information, see Loading Data from Amazon S3 in the Amazon Redshift documentation.

For more information on migration agents, see Using Data Migration Agents in the AWS SCT User Guide.

For more information about version 1.0.602, see the AWS Schema Conversion Tool User Guide. You can download SCT from here.

About the Author

Eran Schitzer is a product manager for Database Migration Services at Amazon Web Services.