Amazon SageMaker Inference

Easily deploy and manage machine learning (ML) models for inference

Get started with Amazon SageMaker Inference

Overview

Benefits

Capabilities

Customers

Resources

What’s New

What is Amazon SageMaker Inference?

Amazon SageMaker AI makes it easier to deploy ML models including foundation models (FMs) to make inference requests at the best price performance for any use case. From low latency and high throughput to long-running inference, you can use SageMaker AI for all your inference needs. SageMaker AI is a fully managed service and integrates with MLOps tools, so you can scale your model deployment, reduce inference cost, manage models more effectively in production, and reduce operational burden.

Benefits of SageMaker Model Deployment

Deploy models in production for inference for any use case

SageMaker AI caters to a wide range of inference requirements, from low latency (a few milliseconds) and high throughput (millions of transactions per second) scenarios to long-running inference for use cases such as multilingual text processing, text-image processing, multi-modal understanding, natural language processing, and computer vision. SageMaker AI provides a robust and scalable solution for all your inference needs.

Achieve optimal inference performance and cost

Amazon SageMaker AI offers more than 100 instance types with varying levels of compute and memory to suit different performance needs. To better utilize the underlying accelerators and reduce deployment cost, you can deploy multiple models to the same instance. To further optimize costs, you can use autoscaling, which automatically adjusts the number of instances based on traffic. It shuts down instances when there is no usage, thereby reducing inference costs.

Reduce operational burden using SageMaker MLOps capabilities

As a fully managed service, Amazon SageMaker AI takes care of setting up and managing instances, software version compatibilities, and patching versions. With built-in integration with MLOps features, it helps off-load the operational overhead of deploying, scaling, and managing ML models while getting them to production faster.

Wide range of inference options

Real-Time Inference

Real-time, interactive, and low latency predictions for use cases with steady traffic patterns. You can deploy your model to an endpoint that is fully managed and supports autoscaling.

Learn more

Serverless Inference

Low latency and high throughput for use cases with intermittent traffic patterns. Serverless endpoints automatically launch compute resources and scale them in and out depending on traffic, eliminating the need to choose instance types or manage scaling policies.

Learn more

Asynchronous Inference

Low latency for use cases with large payloads (up to 1 GB) or long processing times (up to one hour), and near real-time latency requirements. Asynchronous Inference helps save costs by autoscaling the instance count to zero when there are no requests to process.

Learn more

Batch Transform

Offline inference on data batches for use cases with large datasets. With Batch Transform, you can preprocess datasets to remove noise or bias, and associate input records with inferences to help with result interpretation.

Learn more

Scalable and cost-effective inference options

Single-model endpoints

One model on a container hosted on dedicated instances or serverless for low latency and high throughput.

Learn more

Multiple models on a single endpoint

Host multiple models to the same instance to better utilize the underlying accelerators, reducing deployment costs by up to 50%. You can control scaling policies for each FM separately, making it easier to adapt to model usage patterns while optimizing infrastructure costs.

Learn more

Serial inference pipelines

Multiple containers sharing dedicated instances and executing in a sequence. You can use an inference pipeline to combine preprocessing, predictions, and post-processing data science tasks.

Learn more

Support for most machine learning frameworks and model servers

Amazon SageMaker inference supports built-in algorithms and prebuilt Docker images for some of the most common machine learning frameworks such as TensorFlow, PyTorch, ONNX, and XGBoost. If none of the pre-built Docker images serve your needs, you can build your own container for use with CPU backed multi-model endpoints. SageMaker inference supports most popular model servers such as TensorFlow Serving, TorchServe, NVIDIA Triton, AWS multi-model server.

Amazon SageMaker AI offers specialized deep learning containers (DLCs), libraries, and tooling for model parallelism and large model inference (LMI), to help you improve performance of foundational models. With these options, you can deploy models including foundation models (FMs) quickly for virtually any use case.

Learn More