Enhanced performance for Amazon Bedrock Custom Model Import

You can now achieve significant performance improvements when using Amazon Bedrock Custom Model Import, with reduced end-to-end latency, faster time-to-first-token, and improved throughput through advanced PyTorch compilation and CUDA graph optimizations. With Amazon Bedrock Custom Model Import you can to bring your own foundation models to Amazon Bedrock for deployment and inference at scale.

These performance enhancements typically come with model initialization overhead that could impact container cold-start times. Amazon Bedrock addresses this with compilation artifact caching. This innovation delivers performance improvements while maintaining existing cold-start performance metrics that customers expect from CMI.

When deploying models with these optimizations, customers will experience a one-time initialization delay during the first model startup, but each subsequent model instance will spin up without this overhead—balancing performance with fast startup times during scaling.

In this post, we introduce how to use the improvements in Amazon Bedrock Custom Model Import.

How the optimization works

The inference engine caches compilation artifacts, removing repeated computational work at startup. When the first model instance starts, it generates compilation artifacts including optimized computational graphs and kernel configurations. These artifacts are stored and reused by later instances, so they skip the compilation process and start faster.

The system computes a unique identifier based on model configuration parameters such as batch size, context length, and hardware specifications. This identifier confirms cached artifacts match each model instance’s requirements, so they can be safely reused.

Stored artifacts include integrity verification to detect corruption during transfer or storage. If corruption occurs, the system clears the cache and regenerates artifacts. Models remain available during this process.

Performance improvements

We tested performance across different model sizes and workload patterns. The benchmarks compared models before and after the compilation caching optimizations, measuring key inference metrics under various concurrency levels from 1 to 32 concurrent requests.

Technical implementation: Compilation caching architecture

With a compilation caching architecture, performance improves because the system no longer repeats computational work at startup. When the first instance of a model starts, the inference engine performs several computationally intensive operations:

1. Computational Graph Optimization: The engine analyzes the model’s neural network architecture and generates an optimized execution plan tailored to the target hardware. This includes operator fusion, memory layout optimization, and identifying opportunities for parallel execution.
2. Kernel Compilation: GPU kernels are compiled and optimized for the specific model configuration, batch size, and sequence length. This compilation process generates highly optimized CUDA code that maximizes GPU utilization.
3. Memory Planning: The engine determines optimal memory allocation strategies, including tensor placement and buffer reuse patterns that minimize memory fragmentation and data movement.

Previously, each new model instance repeated these operations independently, consuming significant initialization time. With compilation caching, the first instance generates these artifacts and helps store them securely. Subsequent instances retrieve and reuse these pre-compiled artifacts, bypassing the compilation phase entirely. The system uses a configuration-based identifier (incorporating model architecture, batch size, context length, and hardware specifications) to make sure cached artifacts match exactly with instance requirements, maintaining correctness while facilitating consistent optimized performance across the instances. The system includes checksum verification to detect corrupted cache files. If verification fails, the system automatically falls back to full compilation, facilitating reliability while maintaining the performance benefits when cache is available.

Benchmarking set up

We benchmarked under conditions that mirror production environments:

Test configuration: Each benchmark run deployed a single model copy per instance without auto-scaling enabled. This isolated configuration makes sure that performance measurements reflect the true capabilities of the optimization without interference from scaling behaviors or resource contention between multiple model copies. By maintaining this controlled environment, we can attribute performance improvements directly to the compilation caching enhancements rather than infrastructure scaling effects.

Workload patterns: We evaluated two representative I/O patterns that span common use cases:

• 1000/250 tokens (1000 input, 250 output): Represents medium-length prompts with moderate response lengths, typical of conversational AI applications, code completion tasks, and interactive Q&A systems
• 2000/500 tokens (2000 input, 500 output): Represents longer context windows with more substantial responses, common in document analysis, detailed code generation, and comprehensive content creation tasks

We chose these patterns because latency varies with the input-to-output ratio. across different token distributions, as latency characteristics can vary significantly based on the ratio of input processing to output generation.

Concurrency levels: Tests were conducted at six concurrency levels (1, 2, 4, 8, 16, 32 concurrent requests) to evaluate performance under varying load conditions. This progression follows powers of two, allowing us to observe how the system scales from single-user scenarios to moderate multi-user loads. The concurrency testing reveals whether optimizations maintain their benefits under increased load and helps identify the potential bottlenecks that emerge at higher request volumes.

Metrics: We captured comprehensive latency statistics across the test runs, including minimum, maximum, average, P50 (median), P95, and P99 percentile values. This complete statistical distribution provides insights into both typical performance and tail latency behavior. The charts in the following section show average latency, which gives a balanced view of overall performance. The full statistical breakdown is available in the accompanying data tables for readers interested in deeper analysis of latency distributions.

Performance metrics definitions

We measured the following performance metrics:

• Time to First Token (TTFT) – The time elapsed from when a request is submitted until the model generates and returns the first token of its response. This metric is critical for user experience in interactive applications, as it determines how quickly users see the model begin responding. Lower TTFT values create a more responsive feel, especially important for streaming applications where users are waiting for the response to begin.
• End-to-End Latency (E2E) – The total time from request submission to complete response delivery, encompassing the processing stages including input processing, token generation, and output delivery. This represents the full wait time for a complete response.
• Throughput – The total number of tokens (both input and output) processed per second across the concurrent requests. Higher throughput means you can serve more users with the same hardware.
• Output Tokens Per Second (OTPS) – The rate at which the model generates output tokens during the response generation phase. This metric specifically measures generation speed and is particularly relevant for streaming applications where users see tokens appearing in real-time. Higher OTPS values result in smoother, faster-appearing text generation, improving the perceived responsiveness of streaming responses.

Inference performance gains

The compilation caching optimizations deliver substantial improvements across the measured metrics, fundamentally helping enhance the user experience and infrastructure efficiency. The following results showcase the performance gains achieved with two representative models, illustrating how the optimizations scale across different model architectures and use cases.

Granite 20B Code Model

As a larger model optimized for code generation tasks, the Granite 20B model demonstrates particularly impressive gains from compilation caching. The following P50 (median) metrics were measured using the 1000 input / 250 output token workload pattern:

• Time to First Token (TTFT): Reduced from 989.9ms to 120.9ms (87.8% improvement). Users see initial responses 8x faster.
• End-to-End Latency (E2E): Reduced from 12,829ms to 5,290ms (58.8% improvement). Complete requests finish in half the time for faster conversation turns.
• Throughput: Increased from 360.5 to 450.8 tokens/sec (25.0% increase). Each instance processes 25% more tokens per second.
• Output Tokens Per Second (OTPS): Increased from 44.8 to 48.3 tokens/sec (7.8% increase). Faster token generation improves streaming response quality.

The following is a comparison of the metrics in the old and new containers for the granite-20b-code-base-8k model, using the average values in each case for an Input/Output pattern of 1000/250 tokens respectively. 

Llama 3.1 8B Instruct Model

The smaller Llama 3.1 8B model, designed for general instruction-following tasks, also shows significant performance improvements that demonstrate the broad applicability of compilation caching across different model architectures. The following P50 (median) metrics were measured using the 1000 input / 250 output token workload pattern:

• Time to First Token (TTFT): Reduced from 366.9ms to 85.5ms (76.7% improvement).
• End-to-End Latency: Reduced from 3,102ms to 2,532ms (18.4% improvement).
• Throughput: Increased from 714.3 tokens/sec to 922.0 tokens/sec (29.1% increase).
• Output Tokens Per Second (OTPS): Increased from 93.9 tokens/sec to 102.4 tokens/sec) (9.1% increase).

The following is a comparison of the metrics in the old and new containers for the llama3.1-8b model, using the average values in each case for an Input/Output pattern of 1000/250 tokens respectively.

The following table shows the full benchmarking metrics for Llama-3.1-8B-Instruct, single model copy, I/O tokens of 2000/500 respectively. Times are in milliseconds and RPS stands for requests per second.

The following table shows the full benchmarking metrics for Llama-3.1-8B-Instruct, single model copy, I/O tokens of 1000/250 respectively. Times are in milliseconds and RPS stands for requests per second.

The following table shows the full benchmarking metrics for granite-20B-code-base-8K, single model copy, I/O tokens of 2000/500 respectively. Times are in milliseconds and RPS stands for requests per second.

The following table shows the full benchmarking metrics for Llama-3.1-8B-Instruct, single model copy, I/O tokens of 1000/250 respectively. Times are in milliseconds and RPS stands for requests per second.

The following table shows the full benchmarking metrics for granite-20B-code-base-8K, single model copy, I/O tokens of 1000/250 respectively. Times are in milliseconds and RPS stands for requests per second.

Performance consistency across load conditions

These improvements remain consistent across different concurrency levels (1-32 concurrent requests), demonstrating the optimization’s effectiveness under varying load conditions. The reduced latency and increased throughput enable applications to serve more users with better response times while using the same infrastructure.

The benefits remain consistent during scaling events. When auto-scaling adds new instances to handle increased traffic, those instances leverage cached compilation artifacts to deliver the same optimized performance. This facilitates consistent inference performance across the instances, maintaining a high-quality user experience during traffic spikes.

Customer impact

These optimizations improve performance during initial deployment, scaling, and instance replacement. The compilation artifact caching makes sure that the performance benefits remain consistent as new instances are added, without requiring each instance to repeat the compilation process.

Chatbots and AI content generators can add capacity faster during peak usage, reducing wait times. Development teams experience shorter deployment cycles when updating models or testing configurations.

Reduced Time to First Token makes applications feel more responsive. Higher Output Tokens Per Second means you can serve more users with existing infrastructure. You can handle larger workloads without adding proportional compute resources.

Faster instance initialization makes auto-scaling more predictable. You can maintain performance during traffic spikes without over-provisioning.

Conclusion

Amazon Bedrock Custom Model Import now delivers substantial improvements in inference performance through compilation artifact caching and advanced optimization techniques. These enhancements reduce time-to-first-token, end-to-end latency, and increase throughput without requiring customer intervention. The compilation artifact caching system makes sure that performance gains remain consistent as your application scales to meet demand.

Existing users can benefit immediately. New users can see enhanced performance from their first deployment. To experience these performance improvements, import your custom models to Amazon Bedrock Custom Model Import today. For implementation guidance and supported model architectures, refer to the Custom Model Import documentation.

About the authors

Nick McCarthy is a Senior Generative AI Specialist Solutions Architect on the Amazon Bedrock team, focused on model customization. He has worked with AWS clients across a wide range of industries — including healthcare, finance, sports, telecommunications, and energy — helping them accelerate business outcomes through the use of AI and machine learning. Outside of work, Nick loves traveling, exploring new cuisines, and reading about science and technology. He holds a Bachelor’s degree in Physics and a Master’s degree in Machine Learning.

Prashant Patel is a Senior Software Development Engineer in AWS Bedrock. He’s passionate about scaling large language models for enterprise applications. Prior to joining AWS, he worked at IBM on productionizing large-scale AI/ML workloads on Kubernetes. Prashant has a master’s degree from NYU Tandon School of Engineering. While not at work, he enjoys traveling and playing with his dogs.

Yashowardhan Shinde is a Software Development Engineer passionate about solving complex engineering challenges in large language model (LLM) inference and training, with a focus on infrastructure and optimization. He has worked across industry and research settings, contributing to building scalable GenAI systems. Yashowardhan has a master’s degree in Machine Learning from the University of California, San Diego. Outside of work, he enjoys traveling, trying out new food, and playing soccer.

Yanyan Zhang is a Senior Generative AI Data Scientist at Amazon Web Services, where she has been working on cutting-edge AI/ML technologies as a Generative AI Specialist, helping customers use generative AI to achieve their desired outcomes. Yanyan graduated from Texas A&M University with a PhD in Electrical Engineering. Outside of work, she loves traveling, working out, and exploring new things.