AWS Architecture Blog
Field Notes: Accelerate Research with Managed Jupyter on Amazon SageMaker
Research organizations across industry verticals have unique needs. These include facilitating stakeholder collaboration, setting up compute environments for experimentation, handling large datasets, and more. In essence, researchers want the freedom to focus on their research, without the undifferentiated heavy-lifting of managing their environments.
In this blog, I show you how to set up a managed Jupyter environment using custom tools used in Life Sciences research. I show you how to transform the developed artifacts into scripted components that can be integrated into research workflows. Although this solution uses Life Sciences as an example, it is broadly applicable to any vertical that needs customizable managed environments at scale.
Overview of solution
This solution has two parts. First, the System administrator of an organization’s IT department sets up a managed environment and provides researchers access to it. Second, the researchers access the environment and conduct interactive and scripted analysis.
This solution uses AWS Single Sign-On (AWS SSO), Amazon SageMaker, Amazon ECR, and Amazon S3. These services are architected to build a custom environment, provision compute, conduct interactive analysis, and automate the launch of scripts.
Walkthrough
The architecture and detailed walkthrough are presented from both an admin and researcher perspective.
Architecture from an admin perspective
In order of tasks, the admin:
- authenticates into AWS account as an AWS Identity and Access Management (IAM) user with admin privileges
- sets up AWS SSO and users who need access to Amazon SageMaker Studio
- creates a Studio domain
- assigns users and groups created in AWS SSO to the Studio domain
- creates a SageMaker notebook instance shown generically in the architecture as Amazon EC2
- launches a shell script provided later in this post to build and store custom Docker image in a private repository in Amazon ECR
- attaches the custom image to Studio domain that the researchers will later use as a custom Jupyter kernel inside Studio and as a container for the SageMaker processing job.
Architecture from a researcher perspective
In order of tasks, the researcher:
- authenticates using AWS SSO
- SSO authenticates researcher to SageMaker Studio
- researcher performs interactive analysis using managed Jupyter notebooks with custom kernel, organizes the analysis into script(s), and launches a SageMaker processing job to execute the script in a managed environment
- the SageMaker processing job reads data from S3 bucket and writes data back to S3. The user can now retrieve and examine results from S3 using Jupyter notebook.
Prerequisites
For this walkthrough, you should have:
- An AWS account
- Admin access to provision and delete AWS resources
- Researchers’ information to add as SSO users: full name and email
Set up AWS SSO
To facilitate collaboration between researchers, internal and external to your organization, the admin uses AWS SSO to onboard to Studio.
For admins: follow these instructions to set up AWS SSO prior to creating the Studio domain.
Onboard to SageMaker Studio
Researchers can use just the functionality they need in Amazon SageMaker Studio. Studio provides managed Jupyter environments with sharable notebooks for interactive analysis, and managed environments for script execution.
When you onboard to Studio, a home directory is created for you on Amazon Elastic File System (Amazon EFS) which provides reliable, scalable storage for large datasets.
Once AWS SSO has been setup, follow these steps to onboard to Studio via SSO. Note the Studio domain id (ex. d-2hxa6eb47hdc) and the IAM execution role (ex. AmazonSageMaker-ExecutionRole-20201156T214222) in the Studio Summary section of Studio. You will be using these in the following sections.
Provision custom image
At the core of research is experimentation. This often requires setting up playgrounds with custom tools to test out ideas. Docker images are an effective[CE1] [BM2] way to package those tools and dependencies and deploy them quickly. They also address another critical need for researchers – reproducibility.
To demonstrate this, I picked a Life Sciences research problem that requires custom Python packages to be installed and made available to a team of researchers as Jupyter kernels inside Studio.
For the custom Docker image, I picked a Python package called Pegasus. This is a tool used in genomics research for analyzing transcriptomes of millions of single cells, both interactively as well as in cloud-based analysis workflows.
In addition to Python, you can provision Jupyter kernels for languages such as R, Scala, Julia, in Studio using these Docker images.
Launch an Amazon SageMaker notebook instance
To build and push custom Docker images to ECR, you use an Amazon SageMaker notebook instance. Note that this is not part of SageMaker Studio and unrelated to Studio notebooks. It is a fully managed machine learning (ML) Amazon EC2 instance inside the SageMaker service that runs the Jupyter Notebook application, AWS CLI, and Docker.
- Use these instructions to launch a SageMaker notebook instance.
- Once the notebook instance is up and running, select the instance and navigate to the IAM role attached to it. This role comes with IAM policy ‘AmazonSageMakerFullAccess’ as a default. Your instance will need some additional permissions.
- Create a new IAM policy using these instructions.
- Copy the IAM policy below to paste into the JSON tab.
- Fill in the values for <region-id> (ex. us-west-2), <AWS-account-id>, <studio-domain-id>, <studio-domain-iam-role>. Name the IAM policy ‘sagemaker-notebook-policy’ and attach it to the notebook instance role.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "additionalpermissions",
"Effect": "Allow",
"Action": [
"iam:PassRole",
"sagemaker:UpdateDomain"
],
"Resource": [
"arn:aws:sagemaker:<region-id>:<AWS-account-id>:domain/<studio-domain-id>",
"arn:aws:iam::<AWS-account-id>:role/<studio-domain-iam-role>"
]
}
]
}
- Start a terminal session in the notebook instance.
- Once you are done creating the Docker image and attaching to Studio in the next section, you will be shutting down the notebook instance.
Create private repository, build, and store custom image, attach to SageMaker Studio domain
This section has multiple steps, all of which are outlined in a single bash script.
- First the script creates a private repository in Amazon ECR.
- Next, the script builds a custom image, tags, and pushes to Amazon ECR repository. This custom image will serve two purposes: one as a custom Python Jupyter kernel used inside Studio, and two as a custom container for SageMaker processing.
- To use as a custom kernel inside SageMaker Studio, the script creates a SageMaker image and attaches to the Studio domain.
- Before you initiate the script, fill in the following information: your AWS account ID, Region (ex. us-east-1), Studio IAM execution role, and Studio domain id.
- You must create four files: bash script, Dockerfile, and two configuration files.
- Copy the following bash script to a file named ‘pegasus-docker-images.sh’ and fill in the required values.
#!/bin/bash
# Pegasus python packages from Docker hub
accountid=<fill-in-account-id>
region=<fill-in-region>
executionrole=<fill-in-execution-role ex. AmazonSageMaker-ExecutionRole-xxxxx>
domainid=<fill-in-Studio-domain-id ex. d-xxxxxxx>
if aws ecr describe-repositories | grep 'sagemaker-custom'
then
echo 'repo already exists! Skipping creation'
else
aws ecr create-repository --repository-name sagemaker-custom
fi
aws ecr get-login-password --region $region | docker login --username AWS --password-stdin $accountid.dkr.ecr.$region.amazonaws.com
docker build -t sagemaker-custom:pegasus-1.0 .
docker tag sagemaker-custom:pegasus-1.0 $accountid.dkr.ecr.$region.amazonaws.com/sagemaker-custom:pegasus-1.0
docker push $accountid.dkr.ecr.$region.amazonaws.com/sagemaker-custom:pegasus-1.0
if aws sagemaker list-images | grep 'pegasus-1'
then
echo 'Image already exists! Skipping creation'
else
aws sagemaker create-image --image-name pegasus-1 --role-arn arn:aws:iam::$accountid:role/service-role/$executionrole
aws sagemaker create-image-version --image-name pegasus-1 --base-image $accountid.dkr.ecr.$region.amazonaws.com/sagemaker-custom:pegasus-1.0
fi
if aws sagemaker list-app-image-configs | grep 'pegasus-1-config'
then
echo 'Image config already exists! Skipping creation'
else
aws sagemaker create-app-image-config --cli-input-json file://app-image-config-input.json
fi
aws sagemaker update-domain --domain-id $domainid --cli-input-json file://default-user-settings.json
Copy the following to a file named ‘Dockerfile’.
FROM cumulusprod/pegasus-terra:1.0
USER root
Copy the following to a file named ‘app-image-config-input.json’.
{
"AppImageConfigName": "pegasus-1-config",
"KernelGatewayImageConfig": {
"KernelSpecs": [
{
"Name": "python3",
"DisplayName": "Pegasus 1.0"
}
],
"FileSystemConfig": {
"MountPath": "/root",
"DefaultUid": 0,
"DefaultGid": 0
}
}
}
Copy the following to a file named ‘default-user-settings.json’.
{
"DefaultUserSettings": {
"KernelGatewayAppSettings": {
"CustomImages": [
{
"ImageName": "pegasus-1",
"ImageVersionNumber": 1,
"AppImageConfigName": "pegasus-1-config"
}
]
}
}
}
Launch ‘pegasus-docker-images.sh’ in the directory with all four files, in the terminal of the notebook instance. If the script ran successfully, you should see the custom image attached to the Studio domain.
Perform interactive analysis
You can now launch the Pegasus Python kernel inside SageMaker . If this is your first time using Studio, you can get a quick tour of its UI.
For interactive analysis, you can use publicly available notebooks in Pegasus tutorial from this GitHub repository. Review the license before proceeding.
To clone the repository in Studio, open a system terminal using these instructions. Initiate $ git clone https://github.com/klarman-cell-observatory/pegasus
- In the directory ‘pegasus’, select ‘notebooks’ and open ‘pegasus_analysis.ipynb’.
- For kernel choose ‘Pegasus 1.0 (pegasus-1/1)’.
- You can now run through the notebook and examine the output generated. Feel free to work through the other notebooks for deeper analysis.
At any point during experimentation, you can share your analysis along with results with your colleagues using these steps. The snapshot that you create also captures the notebook configuration such as instance type and kernel, to ensure reproducibility.
Formalize analysis and execute scripts
Once you are done with interactive analysis, you can consolidate your analysis into a script to launch in a managed environment. This is an important step, if you want to later incorporate this script as a component into a research workflow and automate it.
Copy the following script to a file named ‘pegasus_script.py’.
"""
BSD 3-Clause License
Copyright (c) 2018, Broad Institute
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
import pandas as pd
import pegasus as pg
if __name__ == "__main__":
BASE_DIR = "/opt/ml/processing"
data = pg.read_input(f"{BASE_DIR}/input/MantonBM_nonmix_subset.zarr.zip")
pg.qc_metrics(data, percent_mito=10)
df_qc = pg.get_filter_stats(data)
pd.DataFrame(df_qc).to_csv(f"{BASE_DIR}/output/qc_metrics.csv", header=True, index=False)
The Jupyter notebook following provides an example of launching a processing job using the script in SageMaker.
- Create a notebook in SageMaker Studio in the same directory as the script.
- Copy the following code to the notebook and name it ‘sagemaker_pegasus_processing.ipynb’.
- Select ‘Python 3 (Data Science)’ as the kernel.
- Launch the cells.
import boto3
import sagemaker
from sagemaker import get_execution_role
from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput
region = boto3.Session().region_name
sagemaker_session = sagemaker.session.Session()
role = sagemaker.get_execution_role()
bucket = sagemaker_session.default_bucket()
prefix = 'pegasus'
account_id = boto3.client('sts').get_caller_identity().get('Account')
ecr_repository = 'research-custom'
tag = ':pegasus-1.0'
uri_suffix = 'amazonaws.com'
if region in ['cn-north-1', 'cn-northwest-1']:
uri_suffix = 'amazonaws.com.cn'
processing_repository_uri = '{}.dkr.ecr.{}.{}/{}'.format(account_id, region, uri_suffix, ecr_repository + tag)
print(processing_repository_uri)
script_processor = ScriptProcessor(command=['python3'],
image_uri=processing_repository_uri,
role=role,
instance_count=1,
instance_type='ml.m5.xlarge')
!wget https://storage.googleapis.com/terra-featured-workspaces/Cumulus/MantonBM_nonmix_subset.zarr.zip
local_path = "MantonBM_nonmix_subset.zarr.zip"
s3 = boto3.resource("s3")
base_uri = f"s3://{bucket}/{prefix}"
input_data_uri = sagemaker.s3.S3Uploader.upload(
local_path=local_path,
desired_s3_uri=base_uri,
)
print(input_data_uri)
code_uri = sagemaker.s3.S3Uploader.upload(
local_path="pegasus_script.py",
desired_s3_uri=base_uri,
)
print(code_uri)
script_processor.run(code=code_uri,
inputs=[ProcessingInput(source=input_data_uri, destination='/opt/ml/processing/input'),],
outputs=[ProcessingOutput(source="/opt/ml/processing/output", destination=f"{base_uri}/output")]
)
script_processor_job_description = script_processor.jobs[-1].describe()
print(script_processor_job_description)
output_path = f"{base_uri}/output"
print(output_path)
The ‘output_path’ is the S3 prefix where you will find the results from SageMaker processing. This will be printed as the last line after execution. You can examine the results either directly in S3 or by copying the results back to your home directory in Studio.
Cleaning up
To avoid incurring future charges, shut down the SageMaker notebook instance. Detach image from the Studio domain, delete image in Amazon ECR, and delete data in Amazon S3.
Conclusion
In this blog, I showed you how to set up and use a unified research environment using Amazon SageMaker. Although the example pertained to Life Sciences, the architecture and the framework presented are generally applicable to any research space. They strive to address the broader research challenges of custom tooling, reproducibility, large datasets, and price predictability.
As a logical next step, take the scripted components and incorporate them into research workflows and automate them. You can use SageMaker Pipelines to incorporate machine learning into your workflows and operationalize them.