My Amazon SageMaker notebook instance gets stuck in the Pending state, and then fails

Last updated: 2020-09-24

When I create or start an Amazon SageMaker notebook instance, the instance enters the Pending state. The notebook instance appears to be stuck in this state, and then it fails.

Short description

The Pending status means that SageMaker is creating the notebook instance. If any of the steps in the creation process fail, SageMaker attempts to create the notebook again. This is why a notebook might stay in the Pending state longer than expected. If SageMaker still can't create the notebook instance, the status eventually changes to Failed.

Resolution

Confirm the failure reason

Check the FailureReason response in the DescribeNotebookInstance API. You can also find the failure reason on the SageMaker console:

  • To see a pop-up displaying a shortened version of the failure reason, pause on Failed in the Status column.
  • To see the full failure reason, choose the name of the notebook instance. The failure reason appears at the top of the Notebook instance settings section.

Use the failure reason to troubleshoot the root cause.

Common errors

"fatal: unable to access 'https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/': Failed to connect to github.com port 443: Connection timed out"

This error happens when the networking configuration for the notebook instance doesn't support the domain name or connection for the external Git repository.

Important: Notebook instances that are deployed in a virtual private cloud (VPC) don't automatically inherit custom route tables, such as subnet route tables for VPC peering connections. If you need a custom route table, create a lifecycle configuration script that adds the route on startup. For more information, see Understanding Amazon SageMaker notebook instance networking configurations and advanced routing options.

To confirm that the Git connection is valid and that you can connect to the repository from a notebook instance, create a new notebook instance without an associated Git repository. Then, open the Jupyter console and use a terminal session to run the following commands:

1.    Resolve the hostname of the server:

dig repo_hostname

If the answer section of the output is empty, the notebook wasn't able to resolve the hostname.

2.    If the answer section of the output contains a response, the domain name resolution works. You can then run the following command to test the connection to the hostname:

nc repo_hostname 443

3.    If the connection is refused or times out, verify the VPC security group rules and route tables. If the connection is successful, use git commands to test your credentials:

git pull https://your-git-repo-url

"Lifecycle Configuration failed"

If a lifecycle configuration script runs for longer than five minutes, it fails, and the notebook instance is not created or started. For suggestions on how to decrease script run time, see Customize a notebook instance using a lifecycle configuration script. To troubleshoot issues with the script, check the Amazon CloudWatch logs for the lifecycle configuration:

  • Log group: /aws/sagemaker/NotebookInstances
  • Log stream: notebook-instance-name/LifecycleConfigOnStart or notebook-instance-name/LifecycleConfigOnCreate

"This Notebook Instance type 'ml.m4.xlarge' is temporarily unavailable. We apologize for the inconvenience. Please try again in a few minutes, or try a different instance type."

This error happens when Amazon Elastic Compute Cloud (Amazon EC2) doesn't have enough available capacity for the instance type that you chose. Capacity varies, based on the demand for that instance type in that Region at that time. Try the request again later to see if capacity levels have changed. Or, choose a different instance type.

HTTP 500 internal errors

An HTTP 500 error indicates that an unexpected error occurred while creating the notebook instance. To rule out transient issues, try creating the notebook instance again.


Did this article help?


Do you need billing or technical support?