AWS Machine Learning Blog

Create Amazon SageMaker projects with image building CI/CD pipelines

Amazon SageMaker projects are AWS Service Catalog provisioned products that enable you to easily create end-to-end machine learning (ML) solutions. SageMaker projects give organizations the ability to use templates that bootstrap ML solutions for your users to speed up the start time for ML development.

You can now use SageMaker projects to manage custom dependencies through an image building continuous integration and continuous delivery (CI/CD) pipeline that’s available as a first-party template on Amazon SageMaker Studio. This new capability gives developers the flexibility to make updates to the images you use for training, processing, and inference by changing the container files in your project’s source control repositories, which automatically triggers an image building pipeline. The template uses AWS CodeCommit as the code repository. You can use the newly created images in a SageMaker pipeline for processing, training, and inference.

The new template options are now available via the SageMaker Python SDK or within the Studio IDE, as shown in the following screenshot.

The new template uses AWS CodePipeline to build and push images to Amazon ECR and then trigger a SageMaker pipeline that trains a model and registers it to the SageMaker Model Registry. After it’s in the model registry, the model status can be updated to approved, which triggers the model deployment process via CodePipeline.

Solution overview

The following architectural diagram doesn’t include the CodeCommit repositories for the model building and model deployment code. The focus is on the image building features in the new template.

We use the new MLOps project template for image building CI/CD to provision and configure the following resources, which are discussed in more detail later in this post:

  • SageMaker code repositories – A set of five CodeCommit repositories are created by the project template. Three of these repositories contain the code for building images used for processing, training, and inference. They include seed code that you can change to customize your image. There is a repository for the model training code using a SageMaker pipeline and a repository for the model deployment code using AWS CloudFormation and CodePipeline. Through the UI, you can choose which images you want to build as part of the CI/CD pipeline and only those repositories are created.
  • Amazon ECR repositories – Amazon ECR repositories are created for the training, processing, and inference images.
  • Model build and deploy triggers – Amazon EventBridge rules are created to trigger the model build CodePipeline pipeline on Amazon ECR state changes. This automates the process of triggering a SageMaker pipeline when new versions of containers are built and pushed to Amazon ECR. The model deploy CodePipeline pipeline is configured to be triggered by EventBridge when the status of a model in the model registry changes to approved.
  • MLOps S3 bucket – An Amazon Simple Storage Service (Amazon S3) bucket for the MLOps pipeline is used for inputs and artifacts of your project and pipeline.

All of the provisioning and configuration required to set up the end-to-end CI/CD pipeline using these resources is automatically performed by SageMaker projects.

Now that we’ve covered how the new feature works, let’s walk through the one-time setup tasks followed by using the new templates.

Create a new SageMaker project

To create your SageMaker project, complete the following steps:

  1. On the Studio console, choose SageMaker resources.
  2. On the drop-down menu, choose Projects.
  3. Choose Create project.
  4. For SageMaker project templates, choose MLOps template for image building, model building, and model deployment.
  5. Choose Select project template.
  6. You need to provide several parameters to configure the image building process:
    1. Include processing image building pipeline – This generates an image building CI/CD infrastructure for the processing image. This includes a CodePipeline pipeline, CodeBuild job, Amazon ECR repository, and SageMaker image.
    2. Include inference image building pipeline – This generates an image building CI/CD infrastructure for the inference image. This includes a CodePipeline pipeline, CodeBuild job, Amazon ECR repository, and SageMaker image.
    3. Include training image building pipeline – This generates an image building CI/CD infrastructure for the training image. This includes a CodePipeline pipeline, CodeBuild job, Amazon ECR repository, and SageMaker image.
    4. Environment type – The type of environment used by CodeBuild when building the image.
    5. Compute type – The compute type value for the build environment used by CodeBuild.
    6. Container tag name – The tag for the container image. This is a Docker tag, not an AWS tag.
  7. Choose Create project.

A message appears indicating that SageMaker is provisioning and configuring the resources.

When the project is complete, you receive a successful message.

Your project is now listed on the Projects list.

  1. Clone each repository by choosing clone repo… on the Studio UI.

Five CodeCommit repositories are created by this project template:

  • sagemaker-image-building-cicd-demo-project-p-1jrh7jnlz3lx-processing-imagebuild
  • sagemaker-image-building-cicd-demo-project-p-1jrh7jnlz3lx-inference-imagebuild
  • sagemaker-image-building-cicd-demo-project-p-1jrh7jnlz3lx-modelbuild
  • sagemaker-image-building-cicd-demo-project-p-1jrh7jnlz3lx-training-imagebuild
  • sagemaker-image-building-cicd-demo-project-p-1jrh7jnlz3lx-modeldeploy

After all five repositories are cloned, they’re available in the Studio UI.

Image building repository

In this example, we generated all three image building repositories. In this section, you see the structure of the repository and learn how it can be updated to meet your custom requirements.

Each of the image building repositories follow the same structure, as shown in the following screenshot.

The image that’s created in this repository is a simple XGBoost image, but following this structure, the Dockerfile can be updated to meet the use case being worked on. The codebuild-buildspec.yml file is used to configure CodeBuild so that the image can be built and pushed to Amazon ECR.

You can navigate to the CodeBuild console to see the status of the images that are built.

The CodePipeline pipelines associated with each repository run automatically on project creation. New builds are triggered when changes are pushed to the repository. You can see the images on the Amazon ECR console.

When new code is pushed to any of the image building repositories, the CodeBuild project starts and the new version of the image is built and pushed to Amazon ECR. A set of EventBridge rules are created to automate each step of the ML workflow. In this new template, a rule in EventBridge is created to trigger the model build pipeline when a new container version is pushed to Amazon ECR.

The model build pipeline target starts running the SageMaker pipeline.

Update a Dockerfile

In this section, we demonstrate how an update to a Dockerfile in one of the repositories triggers a CodeBuild process that creates and pushes a new image version to Amazon ECR, and the subsequent ML pipeline that’s launched.

  1. Update the Dockerfile.
  2. Push the code to CodeCommit.
  3. Navigate to CodeBuild to verify the new image version is being built.

The following screenshots contain the logs of the CodeBuild stage that builds to container using the updated Dockerfile pushed to the repository.

The image version being pushed to Amazon ECR triggers the SageMaker pipeline in the model build repository.

The model can be viewed and approved in the model registry similar to the workflows in the other MLOps templates on SageMaker Studio.

Summary

In this post, we walked through the new SageMaker MLOps project template for image building CI/CD. With the structure provided in the template, you can modify the Dockerfiles to meet your use case, create a custom template with more image building repositories, or create custom rules for the automatic pipeline triggering. Try it out and let us know if you have any questions in the comments section!

About the Author

Kirit Thadaka is an ML Solutions Architect working in the Amazon SageMaker Service SA team. Prior to joining AWS, Kirit spent time working in early stage AI startups followed by some time consulting in various roles in AI research, MLOps, and technical leadership.

Saurabh Trikande is a Senior Product Manager for Amazon SageMaker Inference. He is passionate about working with customers and is motivated by the goal of democratizing machine learning. He focuses on core challenges related to deploying complex ML applications, multi-tenant ML models, cost optimizations, and making deployment of deep learning models more accessible. In his spare time, Saurabh enjoys hiking, learning about innovative technologies, following TechCrunch, and spending time with his family.