My AWS Glue job is taking a long time to start

Last updated: 2019-09-24

How do I avoid cold starts for AWS Glue jobs?

Short Description

When you submit an ETL job, AWS Glue creates resources based on number of data processing units (DPUs) that you specify. This resource provisioning phase can take 10-15 minutes. When the resources are running, AWS Glue executes the ETL script.

Resolution

After executing a job, AWS Glue preserves the resources in a warm pool for a short period of time. This means that if you submit another job shortly after the first one, the second job starts faster. To reduce the time required for resource provisioning, run a warming job before the actual job run. The resources will be ready to go when you submit the primary job.

Note: To use resources from the warm pool, the second job must use the same AWS account, AWS Identity and Access Management (IAM) role, subnet, and security group as the first job.


Did this article help you?

Anything we could improve?


Need more help?