Skip to main content

AWS Glue

AI Help in AWS Glue

Amazon Q data integration in AWS Glue enables you to build data integration pipelines using natural language. Describe your intent through a chat interface and AWS Glue will generate a complete job. You can test the job and put it into production with a single click. AWS Glue now provides additional generative AI capabilities to modernize your Spark jobs and accelerate troubleshooting, reducing the time spent on undifferentiated tasks. You do not need to be an Apache Spark or SQL expert, or wait for an expert to answer your questions. 

Benefits

Integrate data faster

Integrating data across multiple sources can take days or even months. The Amazon Q Developer data integration capability helps you build data integration jobs with minimal data integration knowledge or coding experience. It abstracts manual tasks so you can spend less time on mundane work and more time on analyzing data.

Tell Amazon Q Developer what you need in English and it will return a complete job for you. For example, with the data integration capability, you can ask Amazon Q Developer to “read JSON files from S3, join on ‘accountid’, and load into DynamoDB” and in response, it will return an end-to-end data integration job that can perform that action. You can review the generated job, test it against sample datasets, and move it to production.

Improve developer productivity

Building data integration jobs is just the beginning. Once a job is authored and deployed into production, you must maintain and troubleshoot it. The errors could be related to connection, environment, syntax, validation, or execution. They can occur when you create, test, publish or run your integration jobs. Troubleshooting jobs often requires perusing through log files and logging into monitoring dashboards.

AWS Glue now provides AI-powered capabilities to modernize your Spark jobs and accelerate troubleshooting. Automatically upgrade to newer Spark versions and get intelligent diagnostics when issues occur, reducing debugging time from days to minutes.

Get expert help instantly

AWS Glue provides AI-powered assistance throughout the entire data integration lifecycle. Built with deep domain knowledge of AWS Glue, it provides expert-level guidance for anything data integration related.

Instead of waiting for SMEs within your organization or hiring a consultant, you can reach out to Amazon Q Developer anytime through AWS Glue console, AWS Glue Studio or through API. The Amazon Q Developer data integration capability is optimized for use with other AWS services. You can easily create jobs that extract, transform and load data that is stored in Amazon DynamoDB, Amazon DocumentDB, Amazon Managed Streaming for Apache Kafka, Amazon Kinesis, Amazon Redshift, and Amazon S3.

Use cases

AI for Data Integration

Generate production-ready ETL jobs by simply describing your requirements in plain English. Data engineers and analysts can quickly create data integration workflows without deep Apache Spark expertise. Get complete, runnable code for common scenarios like joining datasets, aggregating data, and loading into various AWS services.

Learn more about Amazon Q Data Integration.

Automatically analyze and upgrade your Spark jobs to newer versions while maintaining functionality. AWS Glue handles the complexity of identifying and updating scripts, configurations, and dependencies, reducing upgrade projects from months to days. This enables you to take advantage of the latest Spark features and security improvements without manual effort.

Learn more about generative AI upgrades for Apache Spark.
Quickly identify and resolve issues in your Spark jobs using AI-powered troublehsooting. AWS Glue analyzes job metadata, execution logs, and configurations to provide immediate root cause analysis and actionable recommendations. This reduces debugging time from days to minutes and eliminates the need to dive deep into documentation or wait for support.

Learn more about generative AI troubleshooting for Apache Spark.