How can I be sure that manually installed libraries persist in Amazon SageMaker if my lifecycle configuration times out when I try to install the libraries?

Last updated: 2020-06-04

When I try to install additional libraries, my lifecycle configuration scripts run for more than five minutes, which causes the Amazon SageMaker notebook instance to time out. How can I resolve this and be sure that my manually installed libraries persist between notebook instance sessions?

Short Description

If a lifecycle configuration script runs for longer than five minutes, it fails, and the notebook instance isn't created or started. There are two ways to resolve this issue:

  • nohup: The nohup command forces the lifecycle configuration script to continue running in the background until the packages are installed. This method is a best practice for less technical users, and is more appropriate as a short-term solution.
  • Create a custom, persistent Conda installation on the notebook instance's Amazon Elastic Block Store (Amazon EBS) volume: Run the on-create script in the terminal of an existing notebook instance. This script uses Miniconda to create a separate Conda installation on the EBS volume (/home/ec2-user/SageMaker/). Then, run the on-start script as a lifecycle configuration to make the custom environment available as a kernel in Jupyter. This method is recommended for more technical users, and it is a better long-term solution.

Resolution

Use one of the following methods to resolve lifecycle configuration timeouts.

nohup

Use the nohup command to force the lifecycle configuration script to continue running in the background even after the five-minute timeout period expires. Example:

#!/bin/bash
set -e
nohup pip install xgboost &

The script stops running after the libraries are installed. You aren't notified when this happens, but you can use the ps command to find out if the script is still running.

Note: You can also use the nohup command if your lifecycle configuration script times out in other scenarios, such as when you download large Amazon Simple Storage Service (Amazon S3) objects.

Create a custom, persistent Conda installation on the notebook instance's EBS volume

1.    In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:

vim custom-script.sh

2.    Copy the contents of the on-create script into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment.

Note: The notebook instance must have internet connectivity to download the Miniconda installer and ipykernel.

3.    Mark the script as executable, and then run it. Example:

chmod +x custom-script.sh
./custom-script.sh

4.    When installation is complete, stop the notebook instance.

5.    Download the on-start script.

6.    On the stopped notebook instance, add the on-start script as a lifecycle configuration. This script makes the custom environment available as a kernel in Jupyter every time that you start the notebook instance.

7.    Start the notebook instance, and then install your custom libraries in the custom environment. For example, to install pyarrow:

import sys
!conda install --yes --prefix {sys.prefix} -c conda-forge pyarrow

If you get an error message that says that you need to update Conda, run the following commands. Then, try installing the custom libraries again.

!conda install -p "/home/ec2-user/anaconda3" "conda>=4.8" --yes
!conda install -p "/home/ec2-user/SageMaker/custom-miniconda/miniconda" "conda>=4.8" --yes

If you stop and then start your notebook instance, your custom Conda environment and libraries are still available. You don't have to install them again.