AWS Big Data Blog
Category: AWS Glue
Introducing automatic version upgrades for AWS Glue jobs
Today we’re announcing automatic version upgrades for AWS Glue jobs. Set a job’s version to auto, and AWS Glue checks for breaking changes, asks for your approval, then moves the job to the latest stable version and validates the upgraded runs. Learn how auto upgrade works and how to use it across a fleet of jobs with an AI agent.
Introducing the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL
The Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL is now generally available. It analyzes completed Spark job runs and returns cost or performance optimization recommendations in minutes, each grounded in the job’s own Spark telemetry, so any engineer can tune jobs without deep Spark expertise.
Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick
The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.
Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs
Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.
Build a self-healing ETL pipeline with Amazon Bedrock AgentCore
Build a self-healing ETL pipeline on AWS Glue and Amazon Bedrock AgentCore that automatically corrects bad records using domain knowledge, writes the fixes back to the data lake, and can open pull requests to prevent the same errors from recurring.
How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito
Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.
How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog
How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.
Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue
Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.
Introducing the Workday Data Cloud integration with AWS
Workday Data Cloud now integrates with AWS, giving developers bi-directional, zero-copy access to Workday’s governed people and money data from the AWS analytics and AI services they already use, without brittle pipelines or data duplication.
Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality
AWS Glue Data Quality is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0). Write the same DQDL rulesets you use in AWS Glue directly in your Amazon EMR Spark jobs. This post walks through two patterns: lightweight inline validation and an Apache Iceberg Write-Audit-Publish workflow with managed rulesets.









