Fast model loading for AI inference on Amazon EKS

When you scale an AI inference workload on Kubernetes, every new pod must load model weights into GPU memory before serving its first request. For large language models, that means moving 60–200 GiB from Amazon Simple Storage Service (Amazon S3) to GPU memory. With default Run:ai Model Streamer settings, this takes 80–460 seconds on p5.48xlarge instances.

We investigated where cold-start time actually goes, what causes the bottlenecks, and what configuration changes eliminate most of the wait. Two changes, neither requiring code modifications, cut cold-start time for a 64 GiB model using Run:ai Model Streamer, a supported model loader in vLLM and SGLang. Initial launch dropped from 82 seconds to 65 seconds, and subsequent launches on the same node dropped from 82 seconds to 16 seconds. Results scale further with larger models, as shown in the following results tables.

This post explains what we found and why we made the choices we did. It is not a how-to. For the configuration itself (environment variables, concurrency calculations, YAML manifests, and instance-specific recommendations), see Accelerate model loading on Amazon EKS in the Amazon Elastic Kubernetes Service (Amazon EKS) User Guide.

Where cold-start time actually goes

When you hear “200 GiB model”, the natural assumption is that downloading it is the bottleneck. Big file, network transfer, slow download. We started with the same assumption. The bottleneck in the stack actually varies by model size.

We instrumented the full pod startup path on p5.48xlarge and measured each phase independently. Node launch (Karpenter) and container image pull are addressed by other efforts (Karpenter provisioning and SOCI (Seekable OCI) parallel pull), and CUDA graph capture is a small fixed cost. That left two phases we could improve: weights loading from S3, and torch.compile (PyTorch’s built-in model compiler).

For a 64 GiB model (Qwen3.6-35B-A3B):

• Weights loading (S3 to GPU): ~29s, 35% of model startup time.
• torch.compile: ~53s, 65% of model startup time.

For a 203 GiB model (Llama-4-Scout, TP=4 where TP is tensor parallelism, splitting the model across GPUs):

• Weights loading (S3 to GPU): ~423s, 92% of model startup time.
• torch.compile: ~34s, 8% of model startup time.

The bottleneck flips. For models under ~100 GiB, torch.compile dominates startup. For larger models, weights loading dominates. This happens because torch.compile time stays roughly constant (it depends on model architecture complexity, not parameter count), while weights loading scales linearly with file size.

Both phases happen on every pod start by default: torch.compile recompiles from scratch even though the output is identical every time for the same model and hardware, and weights re-download from S3 even when the previous pod loaded the same files minutes earlier. We optimized both.

EKS Auto Mode already accelerates container image pulls with SOCI parallel pull by default on all NVMe-equipped accelerated instances. Model weights loading and torch.compile were the next targets.

What we changed and why

We evaluated several approaches: building an EKS-specific model loading feature, shipping a DaemonSet for cross-node cache synchronization, pre-staging models to NVMe during node boot, and contributing optimizations to the publicly available Run:ai Model Streamer. We chose config-only tuning plus upstream contributions.

Config-only changes ship faster. Run:ai Model Streamer is already integrated into vLLM and SGLang, the two inference engines that cover the majority of our customer base. Tuning its environment variables requires no code changes, no DaemonSets, and no additional infrastructure. You can apply them today.

Contributing upstream benefits all compute options. These same optimizations work on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), and Amazon SageMaker. Holding them in an EKS-proprietary fork would have lost the automatic integration with inference engine releases and created a maintenance burden.

Regional Amazon S3 offers high throughput at low cost for this workload. There is no data transfer charge from EC2 to in-region S3, which makes the effective throughput cost near zero. Streaming large model files sequentially at high throughput maps well to how regional Amazon S3 is designed.

DaemonSets add UX friction. We prototyped a cross-node cache DaemonSet. It works, but deploying and maintaining it adds complexity that you shouldn’t need for a configuration optimization. We documented it as an advanced option in the user guide.

Decision 1: Recommend a single S3 chunk size instead of “more parallelism”

Run:ai Model Streamer downloads model weight files from S3 by splitting them into chunks and fetching each chunk as a separate HTTP byte-range request. The decision we needed to make: what chunk size should we recommend, and can we give customers a single value that works broadly? (The user guide covers how to set it and how to compute the matching concurrency.) This configuration and environment-variable tuning improves Run:ai Model Streamer loading speed directly, so any inference engine that supports Run:ai Model Streamer can benefit.

Why not smaller chunks for more parallelism?

Our first instinct was to use small chunks (256 MiB) with high concurrency (256 threads). More parallel connections to S3 should mean more throughput, right?

We tested this. The results surprised us:

256 MiB through 4 GiB all perform within 5% of each other. 256 parallel connections provided no benefit over 17. The only exception was 8 GiB (exceeding the ~3.9 GiB shard size), which caused a 56% regression.

Why didn’t smaller chunks help?

Run:ai Model Streamer assigns one worker thread per shard file. Each worker opens a connection to S3 and issues HTTP byte-range requests for its chunks. The key detail: within a single worker, these range requests are serial. The worker sends one request, waits for the full response, then sends the next. It does not pipeline or overlap requests on the same connection.

Here is what that looks like for a model with 3 shard files:

Worker A (shard 1): [range 1]──wait──[range 2]──wait──...──[range 15] (1 connection)
Worker B (shard 2): [range 1]──wait──[range 2]──wait──...──[range 15] (1 connection)
Worker C (shard 3): [range 1]──wait──[range 2]──wait──...──[range 15] (1 connection)
                    ├──── parallel across files ────┤
                    │     serial within each file   │

Splitting a 3.9 GiB shard into 15 × 256 MiB chunks gives you 15 serial requests on one connection. Splitting it into 1 × 4 GiB chunk gives you 1 request on one connection. Both use the same connection. The parallelism comes from workers running on different files simultaneously, not from subdividing one file into more pieces.

The connection pool has 64 slots, but this serial pattern means only 1-2 connections per worker are ever active. Making chunks smaller only created more sequential round-trips without adding actual concurrency.

Why 4 GiB specifically

At 4 GiB, chunk size matches shard size (most SafeTensors shards are 3–5 GiB). One chunk per file, one request per worker, no serial sub-requests. It also requires the lowest concurrency setting. This means simpler configuration and fewer TCP connections.

This collapses the tuning problem to one variable you can derive from a file count in your S3 path. The user guide gives the formula and a lookup table.

Why we added an aggressive timeout and retry

Model loading is gated by the slowest request: all chunks must complete before the model is ready. S3 GET latency has a long tail: roughly 10% of requests take 2-3x longer than the median, and a single stalled connection holds up the entire load.

Rather than wait, we chose to kill and retry slow requests: when a request falls below a minimum speed threshold for a few seconds, it’s killed, and the S3 request, which is based on the AWS Common Runtime (AWS CRT), retries on a fresh connection and typically completes in median time. This follows the Amazon S3 performance best practices (retry slow requests aggressively). The exact timeout and speed-limit values are in the user guide.

Why we recommend Run:ai Model Streamer tuning instead of NVMe preloading

We also benchmarked NVMe preloading with s5cmd. In that setup the model loads in two hops: s5cmd runs in an init container to pull the model from S3 to NVMe, then Run:ai Model Streamer loads it from NVMe to the GPU. We still recommend the configuration and environment-variable tuning solution over NVMe preloading, for the following reasons:

• Two-hop loading is slower than tuned direct streaming. s5cmd takes ~25s to load a 200 GB model from S3 to NVMe, and Model Streamer takes another ~20s to stream it to the GPU, for ~45s total. Our recommended approach, streaming directly from S3 to the GPU with environment tuning, takes ~25s in total. Even though NVMe preloading only needs to be done once per node, NVMe-direct streaming is only ~5s faster than tuned S3 streaming for a 200 GB model, and the gap narrows further as the model gets smaller.
• NVMe instance storage is ephemeral local storage that only exists on certain instance types, and its throughput depends on the PCIe generation the instance uses, up to ~14 GiB/s for Gen5. If the chosen instance has no local NVMe, preloading has to fall back to an Amazon Elastic Block Store (Amazon EBS) volume, which is much slower, at ~1 GiB/s. So NVMe preloading benefits only a limited set of instance types.
• S3 scales up the throughput it serves for a hot prefix (“S3 prefix warmup”), so the more often the same model is streamed from a bucket with the same prefix, the higher the throughput becomes over time. In production, where the same model is pulled repeatedly, direct S3 streaming keeps getting faster, which further shrinks the advantage from preloading.
• Environment and configuration tuning also requires less setup than a model-preloading init container.

The remaining gap

After these configuration changes, weights load in 12-14s at TP=1. At TP>1, each tensor-parallel rank only downloads a subset of files. With fewer files per rank, you get fewer parallel connections, and those serial streams may not saturate the available network interface card (NIC) bandwidth. Closing that gap requires keeping all connections busy simultaneously within each file. That’s a code change to the Run:ai engine, not something achievable through configuration tuning. See “What we found investigating further” later in this post.

Decision 2: Cache torch.compile instead of recompiling on every pod start

torch.compile optimizes the model at runtime, but that compilation runs on every pod start and adds about 53 seconds cold start. Rather than pay that cost repeatedly, we cache the compilation artifacts so subsequent pod starts reuse them instead of recompiling from scratch. The caching environment variables work for torch.compile directly, so any inference engine that supports torch.compile, such as vLLM and SGLang, can benefit.

Why not disable torch.compile?

If torch.compile takes 53 seconds, why not skip it? Because it provides 5-30% inference throughput improvement through kernel fusion, fewer kernel launches, and removing Python from the hot path. Disabling it trades 53s of one-time startup cost for permanently slower inference on every request served by that pod. For a pod that runs for hours serving thousands of requests, that tradeoff favors keeping compilation.

So: how do we avoid re-compilation?

Why the default cache isn’t enough on Kubernetes

vLLM already caches compiled artifacts to local disk. On Kubernetes, pods use ephemeral storage. When a pod terminates (scaling event, out-of-memory (OOM) kill, rolling update), its local filesystem is destroyed and the cache goes with it. The next pod recompiles from scratch. The fix is pointing the cache directories at a persistent volume. No code change, only environment variables and a volume mount (documented in the user guide).

Why hostPath on NVMe instead of a PVC

GPU instances have local NVMe instance store that EKS Auto Mode mounts automatically without additional configuration. This storage is included in the instance cost, with ~30 GB/s read throughput. A PVC backed by Amazon EBS would work but requires additional configuration, incurs EBS cost, reads at ~1 GiB/s, and its ReadWriteOnce access mode means only one pod can mount the volume read-write at a time (so multiple replica pods cannot share the same cache volume). If restored from a snapshot, data is lazy-loaded from S3 on first access, causing unpredictable initial latency. The cache is only ~15 MiB for a 60 GiB model, so read speed isn’t the deciding factor. Simplicity is. The NVMe is already mounted and ready on every GPU node Auto Mode provisions.

Why caching is safe

The compiled artifacts are deterministic for a specific tuple: model architecture, GPU architecture, tensor-parallel degree, and PyTorch/Triton/CUDA versions. Same container image + same GPU type + same TP setting = valid cache. An image update or a change of model or GPU architecture invalidates the cache and triggers exactly one recompilation. This determinism makes caching safe: there’s no risk of serving stale or incorrect compiled kernels. (The user guide lists the full set of parameters that invalidate a cross-node cache.)

Why we didn’t ship cross-node sharing by default

We prototyped a DaemonSet that uploads the 15 MiB torch.compile cache to S3 and pre-loads it on new nodes at boot. This reduces first-pod time on new nodes from 65s to 16s (the recompilation is eliminated even before any pod runs). However, it requires deploying a DaemonSet, configuring an S3 bucket, and managing lifecycle. For teams that scale frequently to new nodes, this is documented as an advanced option in the user guide.

The result

Initial launch compiles in 53s and writes ~15 MiB of artifacts to NVMe. The second pod on the same node loads those artifacts in 4s. For models where torch.compile dominates (under ~100 GiB), this is the single biggest improvement. For larger models where weights loading dominates, the S3 tuning (chunk size and timeout/retry from Decision 1) provides the larger reduction, and the torch.compile cache still eliminates the remaining 30-50s. Together, the two changes compound. It’s not “too good to be true”: the 53s is all compilation work (graph tracing, Triton code generation, CUDA cubin compilation, autotuning), and the 4s is only loading pre-compiled binaries from disk, like loading a shared library. There’s nothing to compute.

The tradeoff we accepted: the first pod on a fresh node still pays the full 53s. Every pod after it on that node gets the 4s start. For most production inference workloads (stable models, infrequent image updates, nodes that run multiple pods over their lifetime), this tradeoff works well.

What we found investigating further

After the config-only improvements shipped, we kept investigating the remaining 12-14s of weights loading to understand what the hardware allows versus what stock Run:ai achieves.

Connection pool utilization is low. We sampled TCP connections during a stock Run:ai model load on g5.12xlarge and found only 1-2 of 64 pool connections in ESTABLISHED state at any time. In a controlled experiment with the same binary, pool configuration, and S3 object, submitting all ranges at once achieved 33-39 Gbps. Submitting one at a time achieved 5.96 Gbps. The NIC is rated at 40 Gbps. The calling pattern, not the network, is the constraint.

TP>1 on PCIe instances leaves bandwidth underutilized. On instances without NVLink (like G-series), Run:ai’s distributed mode downloads a quarter of the model per rank, then broadcasts the full model to all ranks through the NVIDIA Collective Communications Library (NCCL). This broadcast takes 35s for a 61 GiB model over PCIe (1.73 GiB/s), which is 73% of model load time at TP=4. Each rank could instead download only its needed byte ranges directly from S3, eliminating the broadcast.

P instances are already fast enough. On p5.48xlarge with environment tuning, weights load in 12s for a 64 GB model. The 3200 Gbps NIC means S3 bandwidth is nowhere near the limit, so torch.compile dominates and the cache eliminates it. Further S3 optimization on P instances provides diminishing returns.

These are observations, not shipped changes. We’re looking at how to contribute them upstream.

Results summary

The following test results use the vLLM inference engine.

Qwen3-6-35B-A3B (67 GiB, TP=2) on p5.48xlarge. Startup time = weights loading + torch.compile. Each cell shows the breakdown.

Llama-4-Scout-17B-16E-Instruct (203 GiB, TP=4) on p5.48xlarge. Startup time = weights loading + torch.compile. Each cell shows the breakdown.

Each optimization builds on the previous one. The first pod on a fresh node still takes 59s (torch.compile must run once). Every subsequent pod on that node starts in 32s. If you need first-pod optimization on new nodes, you can apply the cross-node cache pre-warming approach documented in the user guide.

Get started

These optimizations require no code changes. Apply the environment variables to your existing vLLM or SGLang deployment and observe the improvement on the next pod restart. The full configuration guide with YAML manifests, concurrency calculations, and instance-specific recommendations (including running on EKS with Karpenter rather than Auto Mode) is available in the Accelerate model loading on Amazon EKS section of the Amazon EKS User Guide.

References and further reading

• Amazon EKS User Guide: Accelerate model loading with Run:ai Model Streamer.
• Run:ai Model Streamer (GitHub).
• vLLM: Loading models with Run:ai Model Streamer.
• Amazon S3 performance best practices.

About the authors