How The Baseten Delivery Network Bdn Makes Cold Starts Fast

How the Baseten Delivery Network (BDN) makes cold starts fast Announcing our Series F . Learn more

Infrastructure

How the Baseten Delivery Network (BDN) makes cold starts fast

We built BDN using a tiered caching system that eliminates upstream dependencies at runtime, making cold starts 2-3x faster and more reliable.

Authors

Gregory Kofman

Ujjwal Sarin

Stephen Day

Last updated April 9, 2026

Share

TL;DR The Baseten Delivery Network (BDN) is an opt-in weight delivery system for model deployments. It reduces runtime dependency on upstream providers by mirroring model weights into securely managed storage at deploy time. BDN caches weights across a three-tier hierarchy — node-local disk, an in-cluster peer cache backed by a consistent hash ring, and mirrored origin — and coordinates downloads so that cold starts stay fast even during thundering-herd scale events. BDN also reduces costs: because BDN delivers weights before the user's container starts, weight transfer doesn't consume billable GPU time. And because data from weight sources is only transferred when changed at deployment time, egress costs stay low.

The problem: Weight transfer is brittle at scale When a new replica starts, model weights must be loaded into GPU memory before it can serve a request. A single checkpoint ranges from ~10 GB (a quantized 7B model) to 100s of gigabytes (a full-precision 70B+ model, or a multi-expert architecture like DeepSeek-R1). Weight transfer ends up taking up most of the cold start time. This could be manageable if weight transfer were fast and reliable, but at scale, it's neither. Three failure modes make weight transfer brittle at scale: Slow pulls: Upstream rate limiting and cross-region latency make download throughput unpredictable. For instance, a Hugging Face download that typically takes 90 seconds can take 10 minutes during a popular model launch.

Upstream fragility: Hugging Face, S3, and GCS experience outages, and a runtime dependency on them means their availability is your availability.

Thundering herd: When an autoscaler fires up 50 replicas simultaneously, all 50 try to pull the same files, multiplying bandwidth consumption and upstream load.

A single cache layer doesn't solve this. A shared NFS volume becomes the bottleneck during burst events. And node-local-only cache helps warm nodes but does nothing for cold ones. BDN addresses all three failure modes with a layered approach: own the source, cache by locality, and deduplicate work across the cluster. ✕ High-level architecture overview of BDN. Each section is described in more detail below; connections are state-dependent and not all are represented here (e.g., “BDN resolver” (in the workload plane) would also link to “mirrored chunks” (in secure storage) if cached weights are not present on the node or cluster). Security checks and pod initiation are also excluded to help maintain clarity of the caching system. Own the source: Push-time mirroring When a user deploys a model with BDN enabled, the mirroring pipeline reads the weights configuration from the Truss config and mirrors files into Baseten-managed secure blob storage. The output is a manifest: a list of files with their content hashes or etags, which serves as the authoritative record for what a deployment needs to start. Files are keyed by content hash or etag, depending on what is available from the user’s weight source. This means identical files across different models (the norm for fine-tunes sharing a base) are mirrored and stored once. The mirroring decision is metadata-based: BDN checks whether it already holds a matching file before transferring any bytes. Subsequent pushes that reference already-mirrored content resolve in a metadata check, not a data transfer. The mirroring workflow can achieve 1-5GB/s through high parallelism: file chunks are distributed across a pool of workers to maximize throughput. ✕ When a user deploys a model, BDN checks whether it already holds matching files (via a metadata check). If the weights are not already mirrored (first-time deployment), BDN mirrors files into Baseten-managed secure blob storage. The output is a manifest containing the list of files with their content hashes or etags. The only time BDN accesses a model’s upstream sources is during deploy-time mirroring; after that, every subsequent cold start reads from Baseten-managed storage. This has two implications: first, upstream outages, rate limits, and credential rotations don't affect serving. Second, egress costs are minimized to genuinely new weight data on subsequent model deployments, not repeated pulls of the same model weights across deployments or scale events. Access control is enforced at the deployment level. While content/etag-addressable storage means identical weights aren't duplicated, a deployment's replicas can only read the specific files enumerated in its manifest. Speeding up weight pulls via multi-tier caching Baseten deployments run in workload planes: isolated clusters across regions and availability zones. Each workload plane has its own cache hierarchy. These caching layers allow BDN to achieve a throughput of >2 GB/s to download weights onto H100 nodes. When a deployment has BDN enabled, the BDN CSI (a Container Storage Interface driver running on each node) fetches the deployment's manifest and resolves each file through three tiers. Tier 1: Node-local disk Each GPU node has a local NVMe SSD for weight caching. If a model's weights have been loaded on this node before by any replica for any deployment, the files are already on disk. Reads happen at NVMe speeds (multiple GB/s) with zero network traffic. Because files are content/etag-addressed, a new deployment sharing most weights with a previous version hits cache for the shared files and only fetches the delta. And, because the BDN CSI delivers weights before the model container starts, weight transfer happens outside the billable compute window. Without a solution like BDN, models load their own weights at runtime. Customers pay for GPU time while waiting for downloads to complete. Tier 2: In-cluster peer cache Each workload plane maintains a distributed cache backed by a consistent hash ring spanning essentially all nodes in the cluster. This is not a passive lookup layer; it actively participates in origin downloads. When a file isn't on local disk and needs to come from origin, it's split into fixed-size...