Overview
The Google Cloud Storage Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from Google Cloud Storage. This connector provides comprehensive access to Google Cloud Storage data, facilitating cloud ETL processes for operational reporting, data governance, and more.
Highlights
- Connect to Google Cloud Storage from AWS Glue Jobs
- Simplify data extracts from Google Cloud Storage
Details
Pricing
- $149.00/month
Vendor refund policy
No Refunds
Legal
Vendor terms and conditions
Content disclaimer
Delivery details
Google Cloud Storage Glue Connector
- Amazon ECS
Container image
Containers are lightweight, portable execution environments that wrap server application software in a filesystem that includes everything it needs to run. Container applications run on supported container runtimes and orchestration services, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). Both eliminate the need for you to install and operate your own container orchestration software by managing and scheduling containers on a scalable cluster of virtual machines.
Version release notes
AWS Glue Connector for Google Cloud Storage reads data from Google Cloud Storage.
Additional details
Usage instructions
Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio
What is the Google Cloud Storage connector for AWS Glue?
The Google Cloud Storage Glue Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from Google Cloud Storage. This connector provides comprehensive access to Google Cloud Storage data, facilitating cloud ETL processes for operational reporting, data governance.
Use the Google Cloud Storage Glue connector
To use the Google Cloud Storage Glue connector in your Glue ETL job, you need to activate the connector first and set connector options in the job. This part shows what options you can set to the connector and how to use it in your Glue ETL job.
Connector options you need to set
secret
(required) the secret name which will contains GSP service account credentials.Database
(required) the database from Glue catalog.table
(required) the table name from Glue catalog you want to pull data(case sensitive).
The recommended approach to setting the connector options
Use AWS Secrets Manager can be used to store username, password and other sensitive information related to source connection.
The IAM Role you will be using in the Glue Job should contain the following policies
AWS Glue Service
- To run the job.
AWS Glue Catalog
- To access database and table from Glue Catalog
Amazon EC2 Container Registry
- To access AWS Container Registry.
Secrets Manager
(optional) - If you use AWS Secrets Manager for connection options.
Using the Google Cloud Storage Connector for AWS Glue
Here are the setup steps for configuring the Google Cloud Storage Connector:
Step 1: Setup IAM Role for policies and secret AWS Secrets Manager
Create database and table in AWS Glue Catalog:
- Create table with location as gcs location. e.g. gs://bucket/folder
- Add Schema with partition column if applicable. (Note: partition column should only be type varchar or String.)
- edit and add partition.pattern as table property in case of partitions e.g. partition.pattern : year=${year_column_name}/month${month_column_name}/
Step 2: Setup IAM Role for policies and secret AWS Secrets Manager
Create role and secret:
- IAM Role for policies as described above.
- Optional - Create secret in AWS Secrets Manager for the connector options described above.
Step 3: Setup Google Cloud Storage connector and a related connection on Glue Studio console
To set up the Google Cloud Storage connector and create a connection for your job:
- Subscribe to product and activate the connector using AWS Glue Studio from the top of this instruction page.
- Enter your connection name and choose "Create connection and active connector".
Step 4: Create a job
To create a job from your connection which is created in the previous step:
- Choose the connection and "create job".
- Select your created connection figure on the visual canvas.
- Add connection options and enter the necessary information.
- Follow the earlier step (Connector options you need to set) to add the connection options
- Or you can create an AWS secret with the connector options and attach with the connection as stated in the above step 2.1
- Enter the job name, choose IAM Role created in step 1, and other properties in the "Job details" tab, and Choose "Save"
Step 5: Save and run the job
Run the job after filling in all parameters and creating the connector job.
Support
Vendor support
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.