AWS Big Data Blog

Introducing automatic version upgrades for AWS Glue jobs

by Keerthi Chadalavada, Chris Kha, Pramod Chunduri, Liyuan Lin, Malinda Malwala, Mohit Saxena, Mukul Prasad, Ria Katoch, Rishabh Nair, Shrey Malpani, Shruti Gupta, Wei Tang, and XiaoRun Yu on in Advanced (300), Announcements, AWS Glue Permalink Share

Many organizations run their Apache Spark pipelines on AWS Glue and each new AWS Glue release means another round of version upgrades. New versions bring better performance and new features, but upgrading takes work. For every job, an engineer reviews breaking changes across Spark, Python, and libraries, then runs the job on the new version and confirms that it produces the same results. Many jobs turn out to need no changes, but the team learns that only after doing the work. Across hundreds of jobs, upgrading becomes a project that keeps coming back, so many jobs stay on older versions.

Today, we’re announcing automatic version upgrades for AWS Glue jobs (auto upgrade). When you set a job’s version to auto, AWS Glue analyzes the job, asks for your approval, and moves the job to the latest stable version. Instead of planning, testing, and tracking each upgrade, your team reviews and approves it.

In this post, you learn how auto upgrade works and how to:

  • Turn on auto upgrade for a job and approve the upgrade in the AWS Glue console.
  • Use an AI agent to assess a fleet of jobs, turn on auto upgrade with email notifications, approve the upgrades, and check the results.

How auto upgrade works

Set a job’s GlueVersion to auto to turn on auto-upgrade. The job keeps running as usual on its current version while AWS Glue checks the job for known breaking changes. AWS Glue doesn’t modify your script. When AWS Glue determines that the job needs no code changes, AWS Glue sends you an eligibility summary with the compatibility results for application code, configuration, and dependencies, and requests your approval. The job stays on its current version until you approve. After you approve, AWS Glue runs the job on the new version and checks the upgraded runs against the job’s earlier runs. AWS Glue then sends you an auto upgrade evaluation summary with the results for correctness, performance, and cost. If the upgraded runs pass the checks, AWS Glue completes the upgrade, and all subsequent runs use the new version. If an upgraded run fails, AWS Glue retries the run on the previous version, so your pipeline keeps running, and then reverts the job. For details, see How an auto upgrade evaluation works in the AWS Glue User Guide.

For jobs whose code must change, use generative AI upgrades for Apache Spark in AWS Glue.

Prerequisites

To follow along, you need the following:

  • A user or role with the IAM permissions for auto upgrade.
  • One or more AWS Glue jobs that meet the following requirements:
    • The job is a Spark ETL job (glueetl) on AWS Glue 2.0 or later.
    • The job’s MaxRetries parameter is set to 1 or higher, and job run queuing is turned off.
    • The job is safe to retry.

For the AI agent walkthrough, you also need the following:

  • An AI agent that supports skills and MCP servers, such as Kiro or Claude Code, with the managing-glue-jobs skill from the AWS Glue samples repository on GitHub.
  • The latest version of the AWS Command Line Interface (AWS CLI) version 2.
  • The spark-upgrade MCP server from Amazon SageMaker Unified Studio, which runs the compatibility assessment. To connect the server, see the skill’s README.
  • For email notifications, the Amazon Simple Notification Service (Amazon SNS) and Amazon EventBridge permissions in the skill’s permissions.md file.

To install the skill, copy the skill folder into your agent’s skills folder.

git clone https://github.com/aws-samples/aws-glue-samples.git
mkdir -p <agent-skills-folder>
cp -R aws-glue-samples/utilities/skills/managing-glue-jobs <agent-skills-folder>/

Walkthrough: Use auto upgrade in the AWS Glue console

In the AWS Glue console, you turn on auto upgrade and approve the upgrade one job at a time.

Create a job with auto upgrade

  1. Open the AWS Glue console, and in the navigation pane, choose ETL jobs.
  2. Create a Spark job, and then choose the Job details tab.
  3. For AWS Glue version, choose Auto.
  4. For Number of retries, enter 1 or higher.
  5. Choose Save.

AWS Glue runs a new auto job on the latest stable version and evaluates the job when the next version is released.

Change an existing job to auto upgrade

  1. In the AWS Glue console, in the navigation pane, choose ETL jobs.
  2. Choose the job, and then choose the Job details tab.
  3. For AWS Glue version, choose Auto, and for Number of retries, enter 1 or higher.
  4. Choose Save.
AWS Glue console Job details tab with AWS Glue version set to Auto for the orders-daily job

Figure 1: The Job details tab for orders-daily, with AWS Glue version set to Auto

Approve or decline the upgrade

When AWS Glue finishes the compatibility checks and finds no blocking issues, the job waits for your approval.

  1. Choose the job, and then choose the Upgrade analysis tab. The analysis status shows Waiting for consent.
  2. In the Upgrade consent required banner, choose eligibility summary, and review the summary.
  3. Choose Approve to upgrade the job, or choose Decline to keep the job on its current version.
AWS Glue console Upgrade analysis tab showing the status Waiting for consent for orders-daily

Figure 2: The Upgrade analysis tab for orders-daily, waiting for your approval

The eligibility summary shows the job, the AWS Glue versions, and the compatibility results.

Eligibility summary for orders-daily showing the AWS Glue versions and compatibility results

Figure 3: The eligibility summary for orders-daily

Check the result

After the evaluation ends, the Upgrade analysis tab shows the auto upgrade evaluation summary. The summary shows the result for performance, correctness, and cost, with the expected range and each upgraded run’s value for every check.

Auto upgrade evaluation summary for orders-daily showing performance, correctness, and cost results

Figure 4: The auto upgrade evaluation summary for orders-daily

Walkthrough: Upgrade a fleet of jobs with an AI agent

When you manage many jobs, an AI agent with the managing-glue-jobs skill assesses, turns on, and approves auto upgrade across your jobs from a few prompts. The agent shows each change and asks for your confirmation before it makes the change.

Diagram of the auto upgrade journey for existing jobs, from assessment through validation

Figure 5: The auto upgrade journey for existing jobs

In this walkthrough, you assess your jobs, turn on auto upgrade, approve the upgrade, and check the result. Between these steps, AWS Glue observes and checks each job, and then upgrades and validates the job. Amazon SNS emails you the eligibility summary and the evaluation summary. The steps that follow match the numbers in Figure 5.

A data team runs three Spark ETL jobs in the US West (Oregon) Region and they want to use AWS Glue version 6.0. The jobs orders-daily and clickstream-agg run on AWS Glue 4.0 and write to Amazon Simple Storage Service (Amazon S3). The job customer-profiles runs on AWS Glue 3.0.

Step 1: Assess your jobs

Ask the agent which of your jobs can use auto upgrade. The assessment does not change anything in your environment, so it’s safe to try first. The agent checks each job against the requirements and runs a compatibility assessment. The agent then lists the jobs that can move to AWS Glue 6.0 and asks you to confirm that each job is safe to retry.

AI agent listing the jobs that can move to AWS Glue 6.0 after assessment

Figure 6: The agent assesses the jobs

To see which job isn’t ready and why, ask the agent. The agent names the setting that blocks the upgrade and the fix.

AI agent naming the setting that blocks a job upgrade and the fix

Figure 7: The agent explains why a job isn’t ready

After you update the setting, set the AWS Glue job customer-profiles to auto. For jobs whose code must change, use generative AI upgrades for Apache Spark in AWS Glue.

Step 2: Turn on auto upgrade and set up email notifications

Ask the agent to turn on auto upgrade and set up email notifications in one prompt. The agent sets up the email notifications and sets GlueVersion to auto for each job. The jobs keep running on AWS Glue 4.0 until you approve an upgrade. To get the email notifications, confirm the subscription as the reply describes.

AI agent turning on auto upgrade and setting up email notifications for the jobs

Figure 8: The agent turns on auto upgrade and sets up email notifications

Step 3: Review and approve the upgrade

AWS Glue observes and checks each job in the background, and then you review and approve the upgrade.

Step 3a: AWS Glue observes and checks

You keep running the jobs on their usual schedule. In the background, AWS Glue observes the runs and analyzes each job without changing it. How long this takes depends on how often each job runs.

When a job is ready, AWS Glue emails you and requests your approval. The email names the job and the AWS Glue versions, and links to the job in the AWS Glue console. You have seven days to approve or decline.

Approval request email naming the job and AWS Glue versions with a link to the console

Figure 9: The approval request email

Step 3b: Review and approve the upgrade

Ask the agent which upgrades are waiting. The agent reads the eligibility summary of each job, and then shows the jobs, their write paths, and what to confirm before you approve.

AI agent showing the jobs waiting for approval and their write paths

Figure 10: The agent shows the upgrades that wait for your approval

Before you approve, confirm that the reply lists every path that each job writes to. Then approve the jobs in one prompt, and name each job that you approve.

AI agent approving the named jobs from the review

Figure 11: The agent approves the named jobs

The agent approves only the jobs that you name, and only jobs from the review that it showed you.

Step 4: Check the result

AWS Glue upgrades and validates each job, and then you check the result.

Step 4a: AWS Glue upgrades and validates

After you approve, subsequent runs of each job use AWS Glue 6.0, and AWS Glue validates the upgraded runs. If an upgraded run fails, AWS Glue retries the run on AWS Glue 4.0 and reverts the job. You don’t need to act in this step.

Step 4b: Check the result

When the evaluation of a job ends, AWS Glue emails you the result. Ask the agent for the result of orders-daily. The agent shows the result of each validation check.

AI agent showing each validation check result for orders-daily

Figure 12: The agent shows the result for orders-daily

Ask the agent about the other job. The agent shows the check that failed and suggests a fix.

AI agent showing the failed check for clickstream-agg and a suggested fix

Figure 13: The agent explains why clickstream-agg didn’t upgrade

Your pipeline continues to run on AWS Glue 4.0. The auto upgrade evaluation summary shows what changed, so you can fix the job before you try again.

Conclusion

In this post, we showed how automatic version upgrades for AWS Glue jobs keep your Apache Spark jobs on the latest stable AWS Glue version. You used an AI agent to assess your jobs and turn on auto upgrade with email notifications. You then reviewed the eligibility summary, approved the upgrades, and checked the results in the auto upgrade evaluation summary.

With auto upgrade, your team no longer needs to plan and run an upgrade project for each job. AWS Glue analyzes each job and checks the upgraded runs against the job’s earlier runs. Your team reviews and approves each upgrade. This way you keep your whole fleet of AWS Glue jobs current instead of upgrading one job at a time.

Some jobs need code changes before they can move to a new version. For those jobs, use generative AI upgrades for Apache Spark in AWS Glue, which proposes and validates the code changes for the new version. After you update the code, set the job’s GlueVersion to auto so that AWS Glue keeps the job current from then on.

Auto upgrade is available today in all commercial AWS Regions. For the list of Regions, see Considerations and limitations in the AWS Glue User Guide. To get started, ask an AI agent with the managing-glue-jobs skill to assess the AWS Glue jobs in your account. Try auto upgrade, and share your feedback in the comments.

Special thanks

A special thanks to everyone who contributed from Engineering and Science to the launch of automatic version upgrades: Ezgi Maden, Akshara Ganapathi, Tina Shao, Raghavendhar Thiruvoipadi Vidyasagar, Shoukat Ghouse, David Zhang, Suthan Phillips, Prasad Nadig, Syed Mahmood, Karthik Seshadri, Pradeep Kumar Nalluri, Adam Rohrscheib, Eric Zhang, Tayo Adedokun, Weston Dunn, Kartik Panjabi, Shubham Mehta, Bryan Liles, Steffen Rochel, Kinshuk Pahare, Paul Meighan, and Rick Sears


About the authors

Keerthi Chadalavada

Keerthi Chadalavada

Keerthi is a Principal Engineer in the AWS analytics organization. She works on generative AI capabilities across AWS Glue, Amazon EMR, and Amazon Athena. Her focus is on agents that make it simpler for customers to build, modernize, and operate their data processing workloads at scale.

Xiaorun Yu

XiaoRun is a Software Development Engineer in the AWS analytics organization. He is working on building scalable and reliable generative AI products to solve real customer issues. Outside of work, Xiaorun enjoys exploring new places in the Bay Area.

Shruti Gupta

Shruti is a Software Development Engineer in AWS Analytics, building generative AI capabilities for data processing. Previously, she worked on Amazon Bedrock, focusing on the availability and resilience of large-scale AI inference. She specializes in highly available distributed systems and reliable generative AI at scale.

Malinda Malwala

Malinda is an Applied Scientist in the Data Processing Agents Science team building DevOps AI agents for AWS Analytics services including Amazon EMR and AWS Glue. His expertise lies at the intersection of formal methods, program analysis, and AI, where he builds reliable, safe, and trustworthy AI agents for enterprise applications.

Pramod Chunduri

Pramod is an Applied Scientist in the Data Processing Agents Science team, where he designs, evaluates, and improves AI agents for AWS Analytics services including Amazon EMR and AWS Glue. His research lies at the intersection of database systems and AI, with a focus on the reliability of AI agents in complex data processing and analytics workflows.

Liyuan Lin

Liyuan Lin is a Software Engineer at AWS Glue, where she works on building generative AI and data integration tools to help customers solve their data challenges. She specializes in developing solutions that combine AI capabilities with data integration workflows, making it easier for customers to manage and transform their data effectively.

Wei Tang

Wei Tang is a Software Development Engineer on the AWS Glue team. She is a strong developer with deep interests in solving recurring customer problems with distributed systems and generative AI. Her focus is on building agentic tools that make it simpler for customers to run, troubleshoot, and modernize their data processing workloads on AWS.

Rishabh Nair

Rishabh is a Software Development Engineer in the AWS analytics organization, where he combines generative AI with distributed systems to build agentic workflows that modernize large-scale data processing. He is passionate about the infrastructure that makes these workflows reliable and scalable for customers.

Chris Kha

Chris is a Software Development Engineer at AWS Glue where he works on creating agentic tools across the AWS Analytics Organization. He specializes in building scalable and reliable fully-managed systems for data processing.

Ria Katoch

Ria is a Software Development Manager in the AWS analytics organization. She leads the team that builds generative AI agents for AWS Glue, Amazon EMR, and Amazon Athena, where the focus is on helping customers keep their data processing workloads current with less effort, so they can spend less time on upgrades and more time getting value from their data.

Shrey Malpani

Shrey Malpani

Shrey is a Senior Product Manager Technical at Amazon Web Services (AWS), where he works at the intersection of distributed data processing and data integration. He is focused on building and scaling data integration and data management capabilities across services like AWS Glue, Amazon EMR, and Amazon Redshift that help customers build AI-ready data platforms for their analytics and machine learning workflows.

Mukul Prasad

Mukul is a Senior Applied Science Manager in AWS Analytics. He leads the Data Processing Agents Science team developing DevOps agents to simplify and optimize the customer journey in using AWS Big Data processing services including Amazon EMR, AWS Glue, Amazon Athena, and Amazon SageMaker Unified Studio.

Mohit Saxena

Mohit leads AWS Glue, the native serverless Apache Spark data integration service, and Agentic AI applications across broader AWS Data Analytics — AWS Glue, Amazon EMR, and Amazon Athena. His teams build systems that simplify how customers build, modernize, troubleshoot and optimize data processing workloads at scale.