Listing Thumbnail

    AWS Glue Google Cloud Storage Connector

     Info
    Easily connect to Google Cloud Storage from AWS Glue
    Listing Thumbnail

    AWS Glue Google Cloud Storage Connector

     Info

    Overview

    The Google Cloud Storage Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from Google Cloud Storage. This connector provides comprehensive access to Google Cloud Storage data, facilitating cloud ETL processes for operational reporting, data governance, and more.

    Highlights

    • Connect to Google Cloud Storage from AWS Glue Jobs
    • Simplify data extracts from Google Cloud Storage

    Details

    Delivery method

    Delivery option
    Google Cloud Storage Glue Connector

    Latest version

    Operating system
    Linux

    Pricing

    AWS Glue Google Cloud Storage Connector

     Info
    Pricing is based on a fixed monthly subscription cost. You pay the same amount each month for unlimited usage of the product. Pricing is prorated, so you're only charged for the number of days you've been subscribed. Subscriptions have no end date and may be canceled any time.

    Subscription cost

     Info
    $149.00/month

    Vendor refund policy

    No Refunds

    Legal

    Vendor terms and conditions

    Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .

    Content disclaimer

    Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.

    Usage information

     Info

    Delivery details

    Google Cloud Storage Glue Connector

    Supported services: Learn more 
    • Amazon ECS
    Container image

    Containers are lightweight, portable execution environments that wrap server application software in a filesystem that includes everything it needs to run. Container applications run on supported container runtimes and orchestration services, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). Both eliminate the need for you to install and operate your own container orchestration software by managing and scheduling containers on a scalable cluster of virtual machines.

    Version release notes

    AWS Glue Connector for Google Cloud Storage reads data from Google Cloud Storage.

    Additional details

    Usage instructions

    Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio 

    What is the Google Cloud Storage connector for AWS Glue?

    The Google Cloud Storage Glue Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from Google Cloud Storage. This connector provides comprehensive access to Google Cloud Storage data, facilitating cloud ETL processes for operational reporting, data governance.

    Use the Google Cloud Storage Glue connector

    To use the Google Cloud Storage Glue connector in your Glue ETL job, you need to activate the connector first and set connector options in the job. This part shows what options you can set to the connector and how to use it in your Glue ETL job.

    Connector options you need to set

    • secret (required) the secret name which will contains GSP service account credentials.
    • Database (required) the database from Glue catalog.
    • table (required) the table name from Glue catalog you want to pull data(case sensitive).

    The recommended approach to setting the connector options

    Use AWS Secrets Manager can be used to store username, password and other sensitive information related to source connection.

    The IAM Role you will be using in the Glue Job should contain the following policies

    AWS Glue Service - To run the job. AWS Glue Catalog - To access database and table from Glue Catalog Amazon EC2 Container Registry - To access AWS Container Registry. Secrets Manager (optional) - If you use AWS Secrets Manager for connection options.

    Using the Google Cloud Storage Connector for AWS Glue

    Here are the setup steps for configuring the Google Cloud Storage Connector:

    Step 1: Setup IAM Role for policies and secret AWS Secrets Manager

    Create database and table in AWS Glue Catalog:

    1. Create table with location as gcs location. e.g. gs://bucket/folder
    2. Add Schema with partition column if applicable. (Note: partition column should only be type varchar or String.)
    3. edit and add partition.pattern as table property in case of partitions e.g. partition.pattern : year=${year_column_name}/month${month_column_name}/

    Step 2: Setup IAM Role for policies and secret AWS Secrets Manager

    Create role and secret:

    1. IAM Role for policies as described above.
    2. Optional - Create secret in AWS Secrets Manager for the connector options described above.

    Step 3: Setup Google Cloud Storage connector and a related connection on Glue Studio console

    To set up the Google Cloud Storage connector and create a connection for your job:

    1. Subscribe to product and activate the connector using AWS Glue Studio from the top of this instruction page.
    2. Enter your connection name and choose "Create connection and active connector".

    Step 4: Create a job

    To create a job from your connection which is created in the previous step:

    1. Choose the connection and "create job".
    2. Select your created connection figure on the visual canvas.
    3. Add connection options and enter the necessary information.
      • Follow the earlier step (Connector options you need to set) to add the connection options
      • Or you can create an AWS secret with the connector options and attach with the connection as stated in the above step 2.1
    4. Enter the job name, choose IAM Role created in step 1, and other properties in the "Job details" tab, and Choose "Save"

    Step 5: Save and run the job

    Run the job after filling in all parameters and creating the connector job.

    Support

    AWS infrastructure support

    AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.

    Similar products

    Customer reviews

    Ratings and reviews

     Info
    0 ratings
    5 star
    4 star
    3 star
    2 star
    1 star
    0%
    0%
    0%
    0%
    0%
    0 AWS reviews
    No customer reviews yet
    Be the first to write a review for this product.