My AWS Glue job is taking a long time to start
Last updated: 2019-09-24
How do I avoid cold starts for AWS Glue jobs?
When you submit an ETL job, AWS Glue creates resources based on number of data processing units (DPUs) that you specify. This resource provisioning phase can take 10-15 minutes. When the resources are running, AWS Glue executes the ETL script.
After executing a job, AWS Glue preserves the resources in a warm pool for a short period of time. This means that if you submit another job shortly after the first one, the second job starts faster. To reduce the time required for resource provisioning, run a warming job before the actual job run. The resources will be ready to go when you submit the primary job.
Note: To use resources from the warm pool, the second job must use the same AWS account, AWS Identity and Access Management (IAM) role, subnet, and security group as the first job.