AWS Big Data Blog

Category: AWS Glue

Introducing automatic version upgrades for AWS Glue jobs

Introducing automatic version upgrades for AWS Glue jobs

by Keerthi Chadalavada, Chris Kha, Pramod Chunduri, Liyuan Lin, Malinda Malwala, Mohit Saxena, Mukul Prasad, Ria Katoch, Rishabh Nair, Shrey Malpani, Shruti Gupta, Wei Tang, and XiaoRun Yu on in Advanced (300), Announcements, AWS Glue Permalink Share

Today we’re announcing automatic version upgrades for AWS Glue jobs. Set a job’s version to auto, and AWS Glue checks for breaking changes, asks for your approval, then moves the job to the latest stable version and validates the upgraded runs. Learn how auto upgrade works and how to use it across a fleet of jobs with an AI agent.

Introducing the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL

Introducing the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL

The Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL is now generally available. It analyzes completed Spark job runs and returns cost or performance optimization recommendations in minutes, each grounded in the job’s own Spark telemetry, so any engineer can tune jobs without deep Spark expertise.

Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

Analyze Anthropic usage, spend, and access with AWS Glue, Amazon S3 Tables, and AWS Quick

The new AWS Glue connector for Anthropic reads Claude usage, cost, and API key data into AWS without custom code. This post shows how to land it in Amazon S3 Tables as Apache Iceberg tables, join it with your business data, query it with Amazon Athena, and visualize AI spend and governance in AWS Quick.

Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

Optimizing PySpark UDFs on AWS Glue 6.0 with native Arrow UDFs

Python UDFs in PySpark can be slow because data is serialized between the JVM and the Python runtime. We benchmark five approaches on AWS Glue 6.0 (standard Python, Arrow-optimized Python, vectorized pandas, native Arrow UDFs, and Spark SQL) across 10M to 100M rows, and separate transport cost from compute cost to explain the ranking.

How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito

How Samsung Ads federated Snowflake Iceberg tables using Amazon Cognito

Learn how Samsung Ads federates Snowflake-managed Apache Iceberg tables into AWS Lake Formation and queries them from Amazon Athena using Amazon Cognito for OAuth2 machine-to-machine authentication, with no second copy of the data to maintain.

How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

How Intuit migrated from Apache Hive Metastore to AWS Glue Data Catalog

How Intuit migrated its data catalog from Apache Hive Metastore to AWS Glue Data Catalog on its journey to an open, secure lakehouse. This post covers the challenges, the phased bidirectional-sync solution, and the benefits: reduced operational overhead, improved data discovery, and tighter integration with AWS analytics services.

Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality

Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality

AWS Glue Data Quality is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0). Write the same DQDL rulesets you use in AWS Glue directly in your Amazon EMR Spark jobs. This post walks through two patterns: lightweight inline validation and an Apache Iceberg Write-Audit-Publish workflow with managed rulesets.