How can I use AWS DMS to migrate data to Amazon S3 in Parquet format?

Last updated: 2019-09-30

How can I use AWS Database Migration Service (AWS DMS) to migrate data in Apache Parquet (.parquet) format to Amazon Simple Storage Service (Amazon S3)?

Resolution

You can use AWS DMS to migrate data to an S3 bucket in Apache Parquet format if you use replication 3.1.3 or a more recent version. The default Parquet version is Parquet 1.0.

Create a target Amazon S3 endpoint from the AWS DMS console and add an event condition action similar to the following:

dataFormat=parquet;

Or create a target Amazon S3 endpoint using the create-endpoint command in the AWS Command Line Interface (AWS CLI):

aws dms create-endpoint --endpoint-identifier s3-target-parque --engine-name s3 --endpoint-type target --s3-settings '{"ServiceAccessRoleArn": "iam-role-arn", "BucketName": "dms-target-s3bucket", "DataFormat": "parquet”}'

Use the following event condition action to specify the Parquet version of output file:

parquetVersion=PARQUET_2_0;

If the output files are still in CSV format, run the describe-endpoints command to see if the value of the DataFormat parameter is in Parquet format:

aws dms describe-endpoints
"DataFormat": "parquet"

If the value of the DataFormat parameter is CSV, then recreate the endpoint.

After you have the output in Parquet format, you can parse the output file by installing the Apache Parquet command line tool:

pip install parquet-cli --user

Then, inspect the file format:

parq LOAD00000001.parquet 
 # Metadata 
 <pyarrow._parquet.FileMetaData object at 0x10e948aa0>
  created_by: AWS
  num_columns: 2
  num_rows: 2
  num_row_groups: 1
  format_version: 1.0
  serialized_size: 169

Finally, print the file content:

parq LOAD00000001.parquet --head
   i        c
0  1  insert1
1  2  insert2

Did this article help you?

Anything we could improve?


Need more help?