Baseten Delivery Network Fast Cold Starts Big Models

Introducing the Baseten Delivery Network: Fast cold starts for big models Announcing our Series F . Learn more

Infrastructure

Introducing the Baseten Delivery Network: Fast cold starts for big models

Cut cold starts by 2–3x for models with billions or trillions of parameters, even when burst scaling to 1,000s of replicas.

Authors

Stephen Day

Kazuyoshi Kato

Gregory Kofman

Rachel Rapp

Last updated March 19, 2026

Share

TL;DR Today we're launching the Baseten Delivery Network (BDN), built from the ground up to eliminate cold start bottlenecks for large models at scale. BDN delivers 2–3x faster cold starts through multi-tier caching and single-flight downloads that prevent thundering herd issues during burst scaling. BDN is now available to all organizations on Baseten!

Everybody hates cold starts. In an ideal world, cold start time would be instantaneous; in practice, it can take anywhere from seconds to hours, depending on your infrastructure and model. Spiky traffic, large models, and upstream dependencies only make scale-up times longer and more complicated to resolve. Today, we're announcing the Baseten Delivery Network (BDN): a solution built from the ground up to address cold start issues for large models at scale. In practice, we see 2-3x faster cold starts compared to alternative solutions for customers scaling models with 10s to 100s of billions of parameters. BDN is integrated into the Baseten Inference Stack , and is available today to any customer on Baseten. What makes cold starts so difficult to solve? A cold start is everything that happens between "a new replica is requested" and "that replica is ready to serve traffic." Cold starts are difficult to solve for because they actually encompass ~6 different steps: Hardware provisioning: Before anything else, a node has to be available. We handle provisioning across a global pool of GPUs with our Multi-cloud Capacity Manager (MCM) — we won’t go into the details of provisioning here.

Scheduling: Once hardware is available, the Kubernetes scheduler assigns your pod to a node. For most deployments this is fast, a few seconds at most.

Image pull: Before your model runs, the container image has to land on the node. Inference images are large (typically multiple GB), but this problem is largely solved with proper image caching and loading techniques.

Weight download: This is where things can get expensive, especially for large models. Your model weights have to be transferred from wherever they're stored to the node's local disk. For frontier-scale models, this can easily be 100s of GB of weights.

Container startup: Once the image is pulled and weights are available, the container process starts. Framework initialization, CUDA context setup, configuration loading — this is typically fast, on the order of seconds, and largely fixed by your runtime choices.

Model load: The inference engine (e.g., vLLM, TensorRT-LLM, or SGLang) loads weights from disk into GPU memory, initializes the KV cache, and potentially compiles CUDA graphs. This phase is largely determined by model size and engine configuration. It's an important optimization target, but one that lives in the engine layer vs. infrastructure.

✕ The cold start process includes 6 steps (including hardware provisioning). Downloading model weights (step 3) is one of the most expensive in the series. Of these phases, hardware provisioning and weight download can be the most variable. Weight download specifically is almost entirely a function of where weights live, how fast you can read them, and whether 1 pod or 500 pods are trying to do the same thing simultaneously. For large models, weight download can dominate total cold start time and degrade sharply under load conditions for spikey traffic. This is what BDN is built for. Why weight loading breaks down at scale The naive approach to weight loading is simple. When a pod needs weights, it pulls them from upstream storage: Hugging Face, S3, GCS. There are three problems with this approach. Slow weight pulls A 70B parameter model can carry over 100GB of weights; many frontier models are easily 100s of GB. Pulling that volume from remote storage on every scale-up means replicas spend most of their cold start time waiting for a network transfer to finish from wherever they happen to live upstream. Upstream fragility When your weight download depends on a third-party service, their uptime becomes your uptime. A model hub outage, a rate limit, an expired credential, or a throttled bucket can make it impossible to download weights, leaving your pods unable to start. The thundering herd When your autoscaler spins up 50 or 100 replicas simultaneously (which is exactly what happens during a traffic burst or a new model rollout), every one of those pods races to the same upstream source for the same hundreds of gigabytes of data at the same time. Object storage has bandwidth limits and rate limits. This becomes a traffic jam: replicas fight each other for download bandwidth, and load times spike from minutes to potentially hours. ✕ Single-flight download (left) vs. thundering herd (right). On the left, a single coordinator node pulls once from object storage, then fans data out to four pods cleanly; on the right, 50–100 pods simultaneously fire individual requests at the same object storage source, saturating bandwidth and turning the download into a traffic jam. Why common solutions don't fully solve weight loading for cold starts Several approaches have emerged to address these issues. They target the symptoms, not the problem. The most common pattern is to put a caching layer between your pods and your upstream object storage. Downloads get faster because frequently used weights live on faster local storage rather than being sent over the network every time. But two problems persist. First, you still have to deal with the thundering herd problem. If 100 pods on 20 nodes all request the same model simultaneously, and your cache processes them independently, this can still cause contention. Second, there's the question of what happens on the first pull. If you’re still fetching weights from upstream object storage, you still have a dependency on its availability, bandwidth limits, and rate limits. Aside from these issues, it can be hard to gauge the impact of solutions that take this approach without a lot of context on the specific infrastructure setup and model. Metrics like...