Posted On: Jan 21, 2022
Amazon SageMaker Pipelines, a fully managed service that allows customers to define and orchestrate their model building steps as workflows, now allows customers to set concurrency limits on the number of steps which can be executed in parallel.
SageMaker Pipelines provides a variety of steps (e.g. processing, training, register model, callback etc.). These steps allow customers flexibility to define various entities in their ML model building workflow. Previously SageMaker Pipelines would execute a step when dependent steps would successfully complete; this may lead to scenarios when a lot of steps are scheduled for execution in parallel. Concurrency control helps customers to control the number of pipeline steps that can execute in parallel. With the new feature, customers can choose to set a granular concurrency limit for an individual step or the entire pipeline. If the concurrency limit is reached, SageMaker Pipelines will hold-off steps and schedule them when executing steps are successfully completed. With this customers can avoid failure due to resource starvation which may be encountered when multiple steps which require the same resources are scheduled for execution at the same time.
This feature is available in all AWS regions where Amazon SageMaker is available. To get started, create a new SageMaker Pipeline from the Amazon SageMaker SDK or Studio and visit our documentation page on concurrency control.