AWS Glue Google Cloud Storage Connector

Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio 

The Google Cloud Storage Glue Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from Google Cloud Storage. This connector provides comprehensive access to Google Cloud Storage data, facilitating cloud ETL processes for operational reporting, data governance.

To use the Google Cloud Storage Glue connector in your Glue ETL job, you need to activate the connector first and set connector options in the job. This part shows what options you can set to the connector and how to use it in your Glue ETL job.

• secret (required) the secret name which will contains GSP service account credentials.
• Database (required) the database from Glue catalog.
• table (required) the table name from Glue catalog you want to pull data(case sensitive).

Use AWS Secrets Manager can be used to store username, password and other sensitive information related to source connection.

AWS Glue Service - To run the job. AWS Glue Catalog - To access database and table from Glue Catalog Amazon EC2 Container Registry - To access AWS Container Registry. Secrets Manager (optional) - If you use AWS Secrets Manager for connection options.

Here are the setup steps for configuring the Google Cloud Storage Connector:

Step 1: Setup IAM Role for policies and secret AWS Secrets Manager

Create database and table in AWS Glue Catalog:

1. Create table with location as gcs location. e.g. gs://bucket/folder
2. Add Schema with partition column if applicable. (Note: partition column should only be type varchar or String.)
3. edit and add partition.pattern as table property in case of partitions e.g. partition.pattern : year=${year_column_name}/month${month_column_name}/

Step 2: Setup IAM Role for policies and secret AWS Secrets Manager

Create role and secret:

1. IAM Role for policies as described above.
2. Optional - Create secret in AWS Secrets Manager for the connector options described above.

Step 3: Setup Google Cloud Storage connector and a related connection on Glue Studio console

To set up the Google Cloud Storage connector and create a connection for your job:

1. Subscribe to product and activate the connector using AWS Glue Studio from the top of this instruction page.
2. Enter your connection name and choose "Create connection and active connector".

Step 4: Create a job

To create a job from your connection which is created in the previous step:

1. Choose the connection and "create job".
2. Select your created connection figure on the visual canvas.
3. Add connection options and enter the necessary information.
   • Follow the earlier step (Connector options you need to set) to add the connection options
   • Or you can create an AWS secret with the connector options and attach with the connection as stated in the above step 2.1
4. Enter the job name, choose IAM Role created in step 1, and other properties in the "Job details" tab, and Choose "Save"

Step 5: Save and run the job

Run the job after filling in all parameters and creating the connector job.