ML Governance with Amazon SageMaker
Simplify access control and enhance transparency
Generate customized roles that allow machine learning (ML) practitioners to start working with SageMaker faster.
Streamline model documentation and provide visibility into key assumptions, characteristics, and artifacts from conception to deployment.
Quickly audit and troubleshoot performance for all models, endpoints, and model monitoring jobs through a unified view.
Track deviations from expected model behavior, as well as missing or inactive monitoring jobs, with automated alerts.
Amazon SageMaker provides purpose-built governance tools to help you implement ML responsibly. With Amazon SageMaker Role Manager, administrators can define minimum permissions in minutes. Amazon SageMaker Model Cards makes it easier to capture, retrieve, and share essential model information, such as intended uses, risk ratings, and training details, from conception to deployment. Amazon SageMaker Model Dashboard keeps you informed on model behavior in production, all in one place.
Watch this video to learn how to improve visibility into your ML models with SageMaker.
How it works
ML governance with Amazon SageMaker uses SageMaker Role Manager, SageMaker Model Cards, and SageMaker Model Dashboard to help you simplify access control and enhance transparency over your ML projects.
Amazon SageMaker includes three main ML governance features: Amazon SageMaker Role Manager, Amazon SageMaker Model Cards, and Amazon SageMaker Model Dashboard. The first section of the diagram is titled “ML governance with Amazon SageMaker.” The description says “Simplify access control and enhance transparency over your ML projects.”
The second section shows the onboarding process, which involves SageMaker Role Manager. SageMaker Role Manager defines minimum permissions in minutes by simplifying permission setup for ML tasks and autogenerating custom IAM policies.
The third section is a part of the model lifecycle, which covers preparing, building, training, deploying, and monitoring models. With SageMaker Model Cards, you can centralize and standardize model documentation, autopopulate model details, and export to a PDF. With SageMaker Model Dashboard, you can monitor model performance through a unified view, access model insights in one place, and receive built-in notifications.
SageMaker Model Cards and SageMaker Model Dashboard are integrated with Amazon SageMaker Model Monitor, which you can use to monitor models for drift, and Amazon SageMaker Clarify, which helps you detect bias and explain model behavior.
Key features
Define minimum permissions in minutes with SageMaker Role Manager
Simplify permissions for ML activities
SageMaker Role Manager provides a baseline set of permissions for ML activities and personas through a catalog of prebuilt AWS Identity and Access Management (IAM) policies. ML activities can include data prep and training, and personas can include ML engineers and data scientists. You can keep the baseline permissions or customize them further based on your specific needs.
Automate IAM policy generation
With a few self-guided prompts, you can quickly input common governance constructs such as network access boundaries and encryption keys. SageMaker Role Manager will then generate the IAM policy automatically. You can discover the generated role and associated policies through the AWS IAM console.
Attach your managed policies
To further tailor the permissions to your use case, attach your managed IAM policies to the IAM role that you create with SageMaker Role Manager. You can also add tags to help identify and organize the roles across AWS services.
Streamline model documentation with SageMaker Model Cards
Capture model information
SageMaker Model Cards is a repository for model information in the Amazon SageMaker Console and helps you centralize and standardize model documentation so you can implement ML responsibly. You can autopopulate training details such as input datasets, training environments, and training results to accelerate the documentation process. You can also add details such as the model purpose and performance goals.
Visualize evaluation results
You can attach model evaluation results such as bias and quality metrics to your model card and add visualizations such as charts to gain key insights into model performance.
Share model cards
You can export your model cards to a PDF format to more easily share them with business stakeholders, internal teams, or your customers.
Get unified model monitoring with SageMaker Model Dashboard
Track model behavior
SageMaker Model Dashboard gives you a comprehensive overview of deployed models and endpoints so that you can track resources and model behavior violations in one place. You can monitor model behavior in four dimensions: data quality, model quality, bias drift, and feature attribution drift. SageMaker Model Dashboard monitors behavior through its integration with Amazon SageMaker Model Monitor and Amazon SageMaker Clarify.
The risk rating shown above is for illustrative purposes only and may vary based on input provided by you.
Automate alerts
SageMaker Model Dashboard provides an integrated experience to set up and receive alerts for missing and inactive model monitoring jobs and deviations in model behavior.
The risk rating shown above is for illustrative purposes only and may vary based on input provided by you.
Troubleshoot model deviations
You can further inspect individual models and analyze factors impacting model performance over time. Then you can follow up with ML practitioners to take corrective measures.
Customers
“At United Airlines, we use machine learning (ML) to improve customer experience by providing personalized offers, enabling customers to be ready using Travel Readiness Center. Our use of ML also extends to airport operations, network planning, flight scheduling. As we were coming out of the pandemic, Amazon SageMaker played a critical role in Travel Readiness Center allowing us to handle large volumes of COVID test certificates, vaccine cards using document-based model automation. With Amazon SageMaker’s new governance capabilities, we have increased control and visibility over our machine learning models. SageMaker Role Manager simplifies the user setup process dramatically by providing baseline permissions and ML activities for each persona linked to IAM roles. With SageMaker Model Cards, our teams can proactively capture and share model information for review, and using SageMaker Model Dashboard, we are able to search and view models deployed on MARS—our internal ML platform. With all these new governance capabilities, we are saving significant amount of time and able to scale up.”
Ashok Srinivas, Director of ML Engineering and Ops, United Airlines
“At Capitec, we have a wide range of data scientists across our product lines, building different ML solutions. Our ML engineers manage a centralized modeling platform built on Amazon SageMaker to empower the development and deployment of all these ML solutions. Without any built-in tools, tracking modeling efforts tends towards disjointed documentation and a lack of model visibility. With SageMaker Model Cards, we can track plenty of model metadata in a unified environment, and SageMaker Model Dashboard affords us visibility into the performance of each model. In addition, SageMaker Role Manager simplifies the process of managing access for data scientists in our different product lines. Each of these contribute towards our model governance being sufficient to warrant the trust that our clients place in us as a financial service provider.”
Dean Matter, ML Engineer, Capitec Bank
Resources
View technical documentation to learn how to use the SageMaker ML governance features.
"Improve ML governance w/deep control & transparency in SageMaker” session from AWS re:Invent 2022.
Define customized permissions in minutes with Amazon SageMaker Role Manager.