Amazon SageMaker introduces Amazon S3 based shared storage for enhanced project collaboration

AWS recently announced that Amazon SageMaker now offers Amazon Simple Storage Service (Amazon S3) based shared storage as the default project file storage option for new Amazon SageMaker Unified Studio projects. This feature addresses the deprecation of AWS CodeCommit while providing teams with a straightforward and consistent way to collaborate on project files across the integrated development tools in SageMaker.

This new Amazon S3 storage option provides the following benefits:

• Simplified collaboration – File sharing between project members directly without Git operations
• Universal access – Consistent file access across SageMaker tools (JupyterLab, Query Editor, Visual ETL)
• Clear workspace separation – Built-in personal storage separation with Amazon Elastic Block Store (Amazon EBS) volumes
• Global availability – Available in AWS Regions where SageMaker is supported

Although Amazon S3 is the default option for file storage, you can also use Git version control for more robust source control capabilities.

In this post, we discuss this new feature and how to get started using Amazon S3 shared storage in SageMaker Unified Studio.

Solution overview

When you create a new SageMaker Unified Studio domain, the service automatically configures Amazon S3 storage as your default project storage option. Each project receives a dedicated shared location in Amazon S3, accessible to project members, following the structure [bucket]/[domain-id]/[project-id]/shared/.

SageMaker tools JupyterLab and Code Editor provide the following to users:

• A personal EBS volume for individual work in JupyterLab and Code Editor tools
• A mounted shared folder containing the project’s Amazon S3 shared storage
• Clear separation between personal and shared spaces

The shared storage is accessible across SageMaker integrated development tools:

• JupyterLab and Code Editor show shared files along with personal files
• Query Editor filters for relevant SQL notebooks
• Visual ETL provides direct access to shared extract, transform, and load (ETL) workflows

Files saved to the shared location are immediately visible and available to project members. Users can continue working with personal files in their EBS volumes in tools like JupyterLab and Code Editor and explicitly move files to shared storage when ready to collaborate.If you want to use Git for collaboration, you can continue to do so by integrating projects with your GitHub version control, GitLab version control, or managed Bitbucket repositories.

Migration and version control options

For teams currently using Amazon CodeCommit, existing projects will remain fully functional. New projects will default to Amazon S3 storage. If you want to have version control for Amazon S3 based projects, you can enable versioning in Amazon S3 directly.

Prerequisites

You will need to complete the following prerequisites before you can follow the instructions in the next section:

1. Sign up for an AWS account.
2. Create a user with administrative access.
3. Enable IAM Identity Center in the same AWS Region you want to create your SageMaker Unified Studio domain. Confirm in which Region SageMaker Unified Studio is currently available. Set up your IdP and synchronize identities and groups with IAM Identity Center. For more information, refer to IAM Identity Center Identity source tutorials.

Get started with Amazon S3 shared storage

To begin using Amazon S3 shared storage, complete the following steps:

1. Create a new SageMaker Unified Studio domain.
2. Create a new project (Amazon S3 storage is the default file storage option).
   
3. Open the new project and choose JupyterLab from the Build menu.
   
4. Save the new notebook you just created.
   
5. Rename the file.
   

After the project is saved, project users can view the saved notebook in the Project files section under the S3 path [bucket]/[domain-id]/[project-id]/shared/.

Enable version control using Git

To enable version control using Git, complete the following steps:

1. On the SageMaker console, create a new project profile.
   
2. Provide the necessary details for your project profile.
   
3. In the Project files storage section, the Amazon S3 option is selected by default. To enable version control for the project, you can use existing Git repository connections by selecting Git repository.
   

Use shared storage in Query Editor

To use the shared storage feature in Query Editor, complete the following steps:

1. Choose Query Editor from the Build menu.
   
2. Compose your query, and on the Actions menu, choose Save to save the query to shared storage.
   
3. Navigate back to the Project files section, where you can view the query notebook files under the S3 path [bucket]/[domain-id]/[project-id]/shared/.
   

Use shared storage in Visual ETL flows

To use the shared storage feature in Visual ETL flows, complete the following steps:

1. Choose Visual ETL flows from the Build menu.
   
2. Develop your ETL workflow and save the code to the project.
   
3. Navigate back to the Project files section, where you can view the files under the S3 path [bucket]/[domain-id]/[project-id]/shared/jobs/uploads/<ETL name>.
   

Clean up

Make sure you remove the SageMaker Unified Studio resources to mitigate any unexpected costs. This involves a few steps:

1. Delete the projects.
2. Delete the domain.
3. Delete the S3 bucket named amazon-datazone-AWSACCOUNTID-AWSREGION-DOMAINID

Conclusion

The launch of Amazon S3 shared storage in SageMaker represents another step in simplifying the analytics and machine learning (ML) development experience for our customers. By reducing the complexity of Git operations while maintaining robust collaboration capabilities, teams can now focus on building and deploying analytics and ML solutions faster. The feature is now available in Regions where SageMaker is available.

For detailed information about this feature, including setup instructions and best practices, refer to Unified storage in Amazon SageMaker Unified Studio. Share your feedback on this feature in the comments section.

About the Authors