Overview
The Azure Data Lake Storage Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from Azure Data Lake Storage Gen2 (ADLS), and also load data into Azure ADLS. This connector provides comprehensive access to Azure ADLS, facilitating cloud ETL processes for operational reporting, backup and disaster recovery, data governance, and more.
Highlights
- Connect to Azure Data Lake Storage Gen2 from AWS Glue Jobs
- Simplify data extracts from and loads to Azure Data Lake Storage Gen2
Details
Features and programs
Financing for AWS Marketplace purchases
Pricing
Vendor refund policy
No refunds.
Legal
Vendor terms and conditions
Content disclaimer
Delivery details
Glue 3.0
- Amazon ECS
- Amazon EKS
Container image
Containers are lightweight, portable execution environments that wrap server application software in a filesystem that includes everything it needs to run. Container applications run on supported container runtimes and orchestration services, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). Both eliminate the need for you to install and operate your own container orchestration software by managing and scheduling containers on a scalable cluster of virtual machines.
Version release notes
Azure Data Lake Storage Connector for AWS Glue.
- This version supports AWS Glue 3.0 and AWS Glue 4.0.
- This version supports both read from and write to Azure Data Lake Storage Gen2.
- This version supports 5 file formats, csv/parquet/json/orc/text.
Additional details
Usage instructions
Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio .
Pre-requisite
- An storage account of Data Lake Storage Gen2 in Azure Cloud
- AWS Secrets Manager
Create a new secret for Azure Data Lake Storage in AWS Secrets Manager
We create a secret in AWS Secrets Manager to store the Azure Data Lake Storage credentials.
- Please create a storage account to use with Azure Data Lake Storage Gen2 in advance. Follow the instructions to get account access keys .
- On the Secrets Manager console, choose Store a new secret.
- For Secret type, select Other type of secret.
- Enter key as accountName for ADLSv2 storage account name.
- Enter key as accountKey for ADLSv2 storage account key.
- (Optional)Enter key as container for ADLSv2 container. Could input as job connection option instead of here.
- Leave the rest of the options at their default.
- Choose Next.
- Give a name to the secret adlstorage_credentials.
- Follow through the rest of the steps to store the secret.
Create a custom connection
Select the created Secret name, adlstorage_credentials, in connection edit page. Then save it.
Create a Glue job and set connection options
Create a Glue Job and specify details:
- Create a job of data source or target from this connector, select custom connection. Then input options and values.
- File format connection options, we support 5 different formats, csv/parquet/json/orc/text. The basic options are path, the Azure data lake storage cloud storage URI, e.g. /input/covid-csv-data/. fileFormat, input or output file format, e.g. csv/parquet/json/orc/text
- For each format, there are different connection options supported. CSV: Option name, header, delimiter, compression, Option Value corresponding,true/false, any delimiter char, none/uncompressed/snappy/gzip/lzo/lz4/brotli/zstd, default value is false, , none. PARQUET, ORC, TEXT and JSON: Option name, compression, Option Value, none/uncompressed/snappy/gzip/lzo/lz4/brotli/zstd, default value is none.
- Remember to set the Glue version to be Glue 3.0 on job detail tab.
Support
Vendor support
Please allow 24 hours
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
