Guidance for Low-Latency, High Throughput Model Inference Using Amazon SageMaker
Overview
This Guidance shows how to use Amazon SageMaker to support high-throughput model inferencing workloads like programmatic advertising and real-time bidding (RTB). For instance, your demand-side platform could use machine learning (ML) models to determine whether to place a bid for an advertising campaign and at what price. By using this Guidance, you can cost-effectively scale to millions of requests per second at a low latency.
Note: Before beginning this Guidance, you will need to containerize your models. SageMaker Model Training provides a wide range of built-in algorithms and frameworks (such as for scikit-learn and XGBoost) you can use to train and tune your ML models. Alternatively, you can bring your own script.
How it works
These technical details feature an architecture diagram to illustrate how to effectively use this solution. The architecture diagram shows the key components and their interactions, providing an overview of the architecture's structure and functionality step-by-step.
Deploy with confidence
Everything you need to launch this Guidance in your account is right here
We'll walk you through it
Let's make it happen
Well-Architected Pillars
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
Operational Excellence
Amazon CloudWatch aggregates logs and creates observability metrics and dashboards, providing visualizations to help you identify performance bottlenecks and troubleshoot requests. You can also set up CloudWatch alarms to identify trends that could be problematic and alert you before they impact your application or business. Additionally, you can use AWS CloudTrail , which keeps track of account activity, to enable governance and risk auditing, as well as facilitate the compliance of your AWS account.
Security
The principle of least privilege is the industry best practice for reducing the surface area of security risks. AWS Identity and Access Management (IAM) policies use least-privilege access so that every policy is restrictive to the specific resource and operation. Additionally, to implement security in layers, this Guidance encrypts data in transit and transfers it over HTTPS, and AWS Key Management Service (AWS KMS) keys encrypt data at rest in Amazon S3 buckets. Finally, real-time bidding (RTB) applications access SageMaker endpoints and Amazon S3 only through PrivateLink , enhancing your security posture.
Reliability
All the services used in this Guidance are serverless and can automatically scale horizontally based on workload demand. In the SageMaker inference endpoints, Amazon EC2 Auto Scaling groups launch instances across AZs to provide high availability. Additionally, Amazon S3 supports features like S3 Versioning , which helps you maintain data version control, prevent accidental deletions, and replicate data to the same or a different AWS Region. With the ability to preserve, retrieve, and restore every version of an object stored in Amazon S3 , you can recover from unintended user actions and application failures.
Performance Efficiency
AWS managed services offload infrastructure management and scaling from you so that you can focus on solving your business needs. In this Guidance, SageMaker manages the hosting of your model inference endpoints. It retrieves the models from Amazon S3 buckets at deployment time, then hosts the most optimal implementation runtime containers. By using its inference recommender and a load-testing tool, SageMaker can choose the optimal instance size based on throughput capacity and incurred latencies. This enables it to manage the scaling of the inference compute through load balancers and Amazon EC2 Auto Scaling groups.
Cost Optimization
This Guidance uses serverless technologies and managed services so that you only pay for the resources you consume. You can also select options to further reduce costs. For example, Amazon SageMaker Savings Plans offer a flexible, usage-based pricing model in exchange for a commitment to a consistent amount of usage. You can also store data cost-effectively by choosing from a range of Amazon S3 storage classes built for specific use cases and access patterns. For example, if you use Amazon S3 Intelligent-Tiering for data with changing, unknown, or unpredictable access patterns—such as data lakes, analytics, or new applications—it will automatically optimize costs by moving your data between tiers for frequent, infrequent, and rare access. Additionally, by securing traffic over a private network using PrivateLink , you can reduce data transfer fees.
Sustainability
This Guidance uses serverless technologies that scale up and down to meet demand so that resources don’t consume energy while idle. Additionally, SageMaker endpoints use custom infrastructure that is optimal to the workload demands of model training and inference, helping you achieve more with fewer resources and a lower carbon footprint.
Disclaimer
Did you find what you were looking for today?
Let us know so we can improve the quality of the content on our pages