Posted On: Apr 21, 2022
Auto Scaling in AWS Glue Apache Spark jobs is now generally available. AWS Glue 3.0 can now dynamically scale resources up and down based on the workload. With Auto Scaling, you no longer need to worry about over-provisioning resources for jobs, spend time optimizing the number of workers, or pay for idle workers.
AWS Glue is a serverless data integration service that allows you to schedule and run data integration and extract, transform, and load (ETL) jobs without managing any computing infrastructure. AWS Glue monitors each stage of the job run and turns workers off when they are idle or adds workers if additional parallel processing is possible. For more details, please refer to the Glue Auto Scaling blog post and visit our documentation.
This feature is now available in all commercial AWS Regions where AWS Glue is available.