How do I resolve the ResourceLimitExceeded error in Amazon SageMaker?
Last updated: 2020-08-25
I'm trying to launch an Amazon SageMaker resource, but I'm getting a ResourceLimitExceeded error. How do I determine which service quota I reached?
Each AWS account has default quotas, formerly referred to as limits, for each AWS service. For a list of Amazon SageMaker quotas, see Service quotas. Unless otherwise noted, each quota is Region-specific. You can request an increase for some—but not all—quotas.
Check the ResourceLimitExceeded error to find the API action that triggered the quota exception. Then, find that API action in the following table and review the associated quotas. It's important to review all quotas for the action. Sometimes, increasing the quota for one resource can cause you to exceed the quota for another resource.
Note: The API actions in the following table include all functions for the action (for example, CreateNotebookInstance, UpdateNotebookInstance, StartNotebookInstance, and so on).
|API action||Corresponding service quotas|
|NotebookInstance||Amazon SageMaker notebooks|
|HyperParameterTuningJob||Amazon SageMaker automatic model tuning|
|ProcessingJob||Amazon SageMaker processing|
|TrainingJob||Amazon SageMaker training and managed spot training|
|Endpoint||Amazon SageMaker hosting|
|TransformJob||Amazon SageMaker batch transform|
|LabelingJob||Amazon SageMaker Ground Truth|
Note: On-demand and Spot Instance quotas are tracked and modified separately. For more information, see Service quotas.
After you determine the resource that you need an increase for, determine how much of an increase to request. This information is in the error message. For example, assume that you're creating an endpoint with 18 ml.m5.2xlarge notebook instances in the us-east-1 Region. You get this error:
ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m5.2xlarge for endpoint usage' is 20 Instances, with current utilization of 4 Instances and a request delta of 18 Instances. Please contact AWS support to request an increase for this limit
Here's how to interpret the error message:
- "Account-level service limit": the quota for the resource
- "Current utilization": the number of resources (ml.m5.2xlarge instances, in this case) that you're already using
- "Request delta": the number of resources that you requested
In this example, there are only 16 ml.m5.2xlarge instances left in the quota, but you requested 18. You need two more ml.m5.2xlarge instances. Request a new quota of 22 ml.m5.2xlarge Amazon SageMaker hosting instances in us-east-1:
- Open the AWS Support Center, and then choose Create case.
- Choose Service limit increase.
- Complete and submit the form.
For more information, see AWS service quotas.