Please subscribe to the product from AWS Marketplace and Activate the Glue connector for Glue 3.0 from Glue Studio .

What is the TPC-DS connector for AWS Glue?

This connector generates TPC-DS compliant datasets. To generate the datasets, you don't need any data sources. After writing the datasets on your resource such as Amazon S3 by the Glue ETL job, the dataset can be used for any benchmarking purpose in your workload.

Please refer to http://www.tpc.org/tpcds/  about TPC-DS and http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf  about TPC-DS specification.

Connector options you need to set

You can pass the following options to the connector.

• table (required) - A table name. You can pick up a table from 25 tables. The table list is in https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-tpcds .
• scale (optional, default is 1) - Generated data size. Possible range is 1 to 100000. You can specify the size of generated data. Specifically scale 1 means that all table data will be generated with 1GB. For example, when you specify 7 for scale, 7GB data of all tables will be generated. This scale factor is described in the section 3 in http://tpc.or/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf .
• numPartitions (optional, default is 1) - The maximum number of concurrency to generate table data in parallel. Please set more than 1 for the concurrent processing with Spark.
  • It's recommended that the value of numPartitions parameter be set based on "Number of Workers" of your ETL job. Here's the calculation formula. The following calculation takes consideration in Glue 2.0 and 3.0. Please be aware that the calculation depends on the "Worker type" of your job as follows.
    • G.1X - numPartitions = (Number of Workers - 1) * 4
    • G.2X - numPartitions = (Number of Workers - 1) * 8
  • For example, when your Glue 3.0 job is set to G.1X as the worker type and 10 number of workers, the numPartitions will be calculated by (10 - 1) * 4 = 36.

You can set up the connector by the below steps in AWS Glue Studio.

Using the TPC-DS connector for AWS Glue

Here's the setup steps for using the TPC-DS connector.

1. Setup TPC-DS custom connector and a related connection on Glue Studio console.
2. Create a job. You set connector options and a necessary job parameter.
3. Save and run the job.

Step 1: Setup TPC-DS connector and create a relevant connection

To set up the TPC-DS connector and create a connection for your job:

1. Subscribe the product and Activate the connector using AWS Glue Studio from the top of this instruction page.
2. Enter your connection name and choose "Create connection and active connector". You can optionally add a description and "Network options". For "Connection access", keep it empty.

Step 2: Create a job

To create a job from your connection which is created in the previous step:

1. Choose the connection and "create job".
2. Select your created connection figure on the visual canvas.
3. Add connection options and enter the necessary information. Specifically table option is required, and if needed, you can specify scale and numPartitions options. (e.g.) table = customer, scale = 10, numPartitions = 30
4. Enter job properties in the "Job details" tab, and Choose "Save"

Step 3. Save and run the job

After filling in all parameters and creating the connector job, run the job.