[SEO Subhead]
This Guidance shows how to use Amazon SageMaker to support high-throughput model inferencing workloads like programmatic advertising and real-time bidding (RTB). For instance, your demand-side platform could use machine learning (ML) models to determine whether to place a bid for an advertising campaign and at what price. By using this Guidance, you can cost-effectively scale to millions of requests per second at a low latency.
Note: Before beginning this Guidance, you will need to containerize your models. SageMaker Model Training provides a wide range of built-in algorithms and frameworks (such as for scikit-learn and XGBoost) you can use to train and tune your ML models. Alternatively, you can bring your own script.
Please note: [Disclaimer]
Architecture Diagram
[Architecture diagram description]
Step 1
A consumer application is deployed within a virtual private cloud (VPC) in your AWS account, using Amazon Virtual Private Cloud (Amazon VPC). This application can be hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or as containers running on either Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Container Service (Amazon ECS).
Step 2
The consumer application connects to Amazon SageMaker Real-Time inference servers using VPC endpoints powered by AWS PrivateLink. This means that all API calls happen over the private network of AWS and not the public internet, minimizing the latency of the invocations and improving your security posture.
Step 3
The inference requests are routed through a Network Load Balancer to the SageMaker real-time inference servers. These servers are hosted across multiple Availability Zones (AZs) within an Amazon EC2 Auto Scaling group. This allows the model inference infrastructure to be elastic and highly available. SageMaker real-time inferences provide a choice of Amazon EC2 instance types.
These include Amazon EC2 Inf1 instances based on AWS Inferentia, high-performance machine learning (ML) inference chips designed and built by AWS, and GPU instances, such as Amazon EC2 G4dn. Multiple hosting options, including shadow testing and an inference recommendation feature in the managed service, reduce operational burden and accelerates time to value.
Step 4
Consumer applications and batch applications use Amazon Simple Storage Service (Amazon S3) to store and retrieve data and use it for offline machine learning (ML) training and experiments. Access to Amazon S3 from the VPC is secured through PrivateLink.
Step 5
Data scientists use SageMaker to experiment, build, and train the ML model. Once the model is ready, it is saved in Amazon S3 for the model inference task to load. Access to Amazon S3 from the VPC is again secured through PrivateLink.
Step 1
A consumer application is deployed within a virtual private cloud (VPC) in your AWS account, using Amazon Virtual Private Cloud (Amazon VPC). This application can be hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or as containers running on either Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Container Service (Amazon ECS).
Step 2
The consumer application connects to Amazon SageMaker Real-Time inference servers using VPC endpoints powered by AWS PrivateLink. This means that all API calls happen over the private network of AWS and not the public internet, minimizing the latency of the invocations and improving your security posture.
Step 3
The inference requests are routed through a Network Load Balancer to the SageMaker real-time inference servers. These servers are hosted across multiple Availability Zones (AZs) within an Amazon EC2 Auto Scaling group. This allows the model inference infrastructure to be elastic and highly available. SageMaker real-time inferences provide a choice of Amazon EC2 instance types.
These include Amazon EC2 Inf1 instances based on AWS Inferentia, high-performance machine learning (ML) inference chips designed and built by AWS, and GPU instances, such as Amazon EC2 G4dn. Multiple hosting options, including shadow testing and an inference recommendation feature in the managed service, reduce operational burden and accelerates time to value.
Step 4
Consumer applications and batch applications use Amazon Simple Storage Service (Amazon S3) to store and retrieve data and use it for offline machine learning (ML) training and experiments. Access to Amazon S3 from the VPC is secured through PrivateLink.
Step 5
Data scientists use SageMaker to experiment, build, and train the ML model. Once the model is ready, it is saved in Amazon S3 for the model inference task to load. Access to Amazon S3 from the VPC is again secured through PrivateLink.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Amazon CloudWatch aggregates logs and creates observability metrics and dashboards, providing visualizations to help you identify performance bottlenecks and troubleshoot requests. You can also set up CloudWatch alarms to identify trends that could be problematic and alert you before they impact your application or business. Additionally, you can use AWS CloudTrail, which keeps track of account activity, to enable governance and risk auditing, as well as facilitate the compliance of your AWS account.
-
SecurityRead the Security whitepaper
The principle of least privilege is the industry best practice for reducing the surface area of security risks. AWS Identity and Access Management (IAM) policies use least-privilege access so that every policy is restrictive to the specific resource and operation. Additionally, to implement security in layers, this Guidance encrypts data in transit and transfers it over HTTPS, and AWS Key Management Service (AWS KMS) keys encrypt data at rest in Amazon S3 buckets. Finally, real-time bidding (RTB) applications access SageMaker endpoints and Amazon S3 only through PrivateLink, enhancing your security posture.
-
ReliabilityRead the Reliability whitepaper
All the services used in this Guidance are serverless and can automatically scale horizontally based on workload demand. In the SageMaker inference endpoints, Amazon EC2 Auto Scaling groups launch instances across AZs to provide high availability. Additionally, Amazon S3 supports features like S3 Versioning, which helps you maintain data version control, prevent accidental deletions, and replicate data to the same or a different AWS Region. With the ability to preserve, retrieve, and restore every version of an object stored in Amazon S3, you can recover from unintended user actions and application failures.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
AWS managed services offload infrastructure management and scaling from you so that you can focus on solving your business needs. In this Guidance, SageMaker manages the hosting of your model inference endpoints. It retrieves the models from Amazon S3 buckets at deployment time, then hosts the most optimal implementation runtime containers. By using its inference recommender and a load-testing tool, SageMaker can choose the optimal instance size based on throughput capacity and incurred latencies. This enables it to manage the scaling of the inference compute through load balancers and Amazon EC2 Auto Scaling groups.
-
Cost OptimizationRead the Cost Optimization whitepaper
This Guidance uses serverless technologies and managed services so that you only pay for the resources you consume. You can also select options to further reduce costs. For example, Amazon SageMaker Savings Plans offer a flexible, usage-based pricing model in exchange for a commitment to a consistent amount of usage. You can also store data cost-effectively by choosing from a range of Amazon S3 storage classes built for specific use cases and access patterns. For example, if you use Amazon S3 Intelligent-Tiering for data with changing, unknown, or unpredictable access patterns—such as data lakes, analytics, or new applications—it will automatically optimize costs by moving your data between tiers for frequent, infrequent, and rare access. Additionally, by securing traffic over a private network using PrivateLink, you can reduce data transfer fees.
-
SustainabilityRead the Sustainability whitepaper
This Guidance uses serverless technologies that scale up and down to meet demand so that resources don’t consume energy while idle. Additionally, SageMaker endpoints use custom infrastructure that is optimal to the workload demands of model training and inference, helping you achieve more with fewer resources and a lower carbon footprint.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.