Azure Blob Connector for AWS Glue

Please subscribe to the product from AWS Marketplace Activate the Glue connector from AWS Glue Studio 

To use the Azure Blob Glue connector in your Glue ETL job, you need to activate the connector first and set connector options in the job. This section shows what options you can set to the connector and how to use it in your Glue ETL job.

Connector options

You can pass the following options to the connector.

• accountName (required when alternative authentication option is absent) Name of the storage account.
• accountKey (required when alternative authentication option is absent) Account key for authentication.
• connectionString (required when accountName/accountKey aren't present) Connection string for authentication.
• containerName (required) The container's name where you keep your data.
• format (required) File format that contains data. Value can be either PARQUET or CSV
• path (optional) When not present, this Glue connector will scan the entire container for files to fetch the data.

The recommended approach to setting the connector options

AWS Secrets Manager can store accountName, accountKey, and connectionString to authenticate the client to the source connection.

The IAM Role you will be using in the Glue Job should contain the following policies

AWS Glue Service - To run the job.
Amazon EC2 Container Registry - To access AWS Container Registry.
Secrets Manager (optional) - If you use AWS Secrets Manager for connection options.

Using the Azure Blob Connector for AWS Glue

Here are the setup steps for configuring the Azure Blob Connector:

1. Setup IAM Role for policies and secret AWS Secrets Manager
2. Setup Azure Blob connector and a related connection on the Glue Studio console.
3. Create a job.
4. Save and run the job.

Step 1: Setup IAM Role for policies and secret AWS Secrets Manager

Create role and secret:

1. IAM Role for policies as described above.
2. Optional - Create a secret in AWS Secrets Manager for the above connector options.

Step 2: Setup Azure Blob connector and a related connection on the Glue Studio console

To set up the Azure Blob connector and create a connection for your job:

1. Subscribe to the product and activate the connector using AWS Glue Studio from the top of this instruction page.
2. Enter your connection name and choose "Create connection and active connector". You can optionally add a description, "Connection access" and "Network options". If you have created an AWS secret with the connector options, you can choose your AWS secret from the dropdown under "Connection access".

Step 3: Create a job

To create a job from your connection which is created in the previous step:

1. Choose the connection and "create job".
2. Select your created connection figure on the visual canvas.
3. Add connection options and enter the necessary information.
   • Follow the earlier step (Connector options you need to set) to add the connection options
   • Or you can create an AWS secret with the connector options and attach it with the connection as stated in the above step 2.1
4. Enter the job name, choose IAM Role created in step 1, and other properties in the "Job details" tab, and Choose "Save"

Step 4: Save and run the job

Run the job after filling in all parameters and creating the connector job.