How can I concatenate Parquet files in Amazon EMR?

Short description

S3DistCp doesn't support concatenation for Parquet files. Use PySpark instead.

Resolution

You can't specify the target file size in PySpark, but you can specify the number of partitions. Spark saves each partition to a separate output file. To estimate the number of partitions that you need, divide the size of the dataset by the target individual file size.

1.    Create an Amazon EMR cluster with Apache Spark installed.

2.    Specify how many executors you need. This depends on cluster capacity and dataset size. For more information, see Best practices for successfully managing memory for Apache Spark applications on Amazon EMR.

$  pyspark --num-executors number_of_executors

3.    Load the source Parquet files into a Spark DataFrame. This can be an Amazon Simple Storage Service (Amazon S3) path or an HDFS path. For example:

df=sqlContext.read.parquet("s3://awsdoc-example-bucket/parquet-data/")

HDFS:

df=sqlContext.read.parquet("hdfs:///tmp/parquet-data/")

4.    Repartition the DataFrame. In the following example, n is the number of partitions.

df_output=df.coalesce(n)

5.    Save the DataFrame to the destination. This can be an Amazon S3 path or an HDFS path. For example:

df_output.write.parquet("URI:s3://awsdoc-example-bucket1/destination/")

HDFS:

df=sqlContext.write.parquet("hdfs:///tmp/destination/")

6.    Verify how many files are now in the destination directory:

hadoop fs -ls "URI:s3://awsdoc-example-bucket1/destination/ | wc -l"

The total number of files should be the value of n from step 4, plus one. The Parquet output committer writes the extra file, called _SUCCESS.