Posted On: Jan 31, 2023
Amazon SageMaker Automatic Model Tuning now supports three new completion criteria to help you customize your tuning jobs based on your desired trade-off between accuracy, cost, and runtime.
With SageMaker Automatic Model Tuning, you can help optimize your machine learning (ML) model by searching for the optimal set of hyperparameter configurations for your dataset using various search strategies. Before this launch, you could choose to specify either max training jobs or a target objective metric to complete the tuning jobs. However, if your tuning job has to be completed before a certain time, it is not trivial to decide how many training jobs to run. You may also not know what target objective metric is reasonable and would rather have the tuning job complete once the objective metric stops improving.
Starting today, SageMaker Automatic Model Tuning offers three additional completion criteria for your tuning jobs. You can now specify MaxRuntimeInSeconds, which will automatically complete a tuning job after a specified amount of time. To stop a tuning job when the best objective is not improving fast enough, you can now also specify MaxNumberOfTrainingJobsNotImproving. Additionally, if you are not sure about what settings to use for these completion criteria, you can now specify a CompleteOnConvergence parameter to automatically stop the tuning job when the objective metric is not improving in subsequent trials. All these new completion criteria allow you to strike your desired balance between cost, runtime and accuracy.
In addition, SageMaker Automatic Model Tuning now includes information in the describe API response to assess these completion criteria. This includes total runtime in seconds, number of training jobs not improving the objective so far, and an indicator of whether the tuning job converged. This information is available regardless of your completion criteria settings, which simplifies your decision making process and helps you determine when to stop your tuning jobs.
The new functionality is now available for SageMaker Automatic Model Tuning in all commercial AWS Regions. To learn more, please visit the API reference guide, the technical documentation, or SageMaker Automatic Model Tuning web page.