Canary rollouts: upgrade models in production without downtime

Canary rollouts: upgrade models in production without downtime Webflow Analyze/Optimize tracking bridge -->

The Open Source AI Stack: models, inference, routers, harnesses & tools →

📈 GLM-5.3 vs. GLM 5.3 Flash on DeepSWE →

🚀 DeepSeek V4 Pro 0813 vs. GPT-5.6 Sol on DeepSWE →

⚡ On-demand B200s now available on Together GPU Clusters →

All blog posts

Inference

Published 9/22/2026

Canary rollouts: upgrade models in production without downtime

Staged traffic ramps, metric gates, and automatic rollback on dedicated inference.

Authors

Siddhant Bharti, Zarni Phyo, Zain Hasan, Nikitha Suryadevara, Ted Cui

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Links in this article

Dedicated Model Inference 101 Autoscaling A/B Testing 📚 Docs: ‍ Start a rollout ‍ ‍ Gate rollouts with metrics ‍ ‍ CLI reference ‍ API reference: Create a rollout ‍

Summary

Rollouts move live traffic from your current model to a new checkpoint in gated steps. Health checks always run before traffic moves; on a canary you can also add metric gates (say p95 latency or error rate) that run after each step. If a gate trips, the rollout pauses at the canary share and you cancel it and run it in reverse. Below we run one for real: a Qwen2.5-7B → Qwen3.5-9B canary whose gate caught a 137% p95 regression at 10% of traffic; we canceled and reversed it with live requests served and none failed.

Swapping models is common practice If you run a model in production, you already know the need to swap in a new checkpoint or a new model family: the open model ecosystem moves fast, and the candidate usually looks great in evals or promises better throughput. You want it in front of every user without hiccups, and a way to rollback if it disappoints. The usual options force a tradeoff: Hard swap: point the endpoint at the new model and every user is exposed at once. If p95 doubles, you find out from your dashboard or, worse, from your customers, and you roll back under pressure onto a cold-started old model. DIY staged cutover: a second deployment plus a script that nudges traffic percentages while you watch Grafana, remembering to scale the old deployment back up before you shift traffic back.

Both of these options put a human in the loop as the safety mechanism. Rollouts move that mechanism into the platform: you describe the source, the target, the steps, and what "healthy" means, and the platform works against this plan at every stage. How rollouts work A rollout migrates traffic between two deployments on the same endpoint: a source (what's serving today) and a target (what you want to serve tomorrow). You pick one of three strategies: Canary: traffic moves in staged percentages you define (say 10% → 50% → 100%; the default ladder is 5% → 25% → 50% → 100%), with a wait period and optional metric checks between steps. Blue-green: one gated 0% → 100% cutover. You can think of this as a single-step canary. Rolling: an in-place, replica-by-replica swap that preserves total capacity. Best when capacity is constrained, especially for same-model config changes that don't need a traffic ramp.

Here's what happens inside every canary step:

We chose this ordering deliberately; each item prevents a class of incidents: The target scales up before any traffic moves. No capacity for the new deployment means no redirected requests: the rollout parks first. The health gate runs before the traffic shifts. Traffic only reaches replicas whose engine is loaded and answering, not merely started. A propagation wait sits between the shift and the drain. Routing caches converge before any source capacity is removed. The source drains after traffic has moved. Capacity leads traffic on the way up; traffic leads capacity on the way down. The wait period and the metric gate come before the step is recorded as complete. A step that regressed is never marked passed.

Through the API or the console, a rollout is created in a PENDING state and does nothing until you explicitly start it (the CLI's rollout command creates and starts in one step). This two-step create/start is intentional because you can create the rollout, review it (or have a teammate review it), and start it when you're actually watching.

Two states in the diagram above deserve a note: PAUSED means you pressed pause. The rollout holds exactly where it is and resumes from the same step. SYSTEM_PAUSED means the platform found something that went wrong, such as a failed metric gate, a capacity shortfall or missing metrics, and stopped to wait for human approval. It pauses, notifies you and waits; canceling is always your call.

There is no FAILED end state that leaves traffic in limbo: a rollout ends COMPLETED (the target serves) or CANCELED (the split is frozen where it was, and you run the rollout in reverse to go back). Anatomy of a step The following is a breakdown of what happens in a single canary step, measured on the run at the end of this post (Qwen2.5-7B → Qwen3.5-9B on one H100 each).

The propagation wait is what keeps stale global routing caches from sending requests to a shrinking source. The wait period is grown to the metric window plus ingestion lag. The cold start dominates the first step; later steps add replicas to a target that is already serving and warm. Choosing a strategy at a glance All three strategies run through the same engine and the same health gates; they differ in how traffic moves, how much extra capacity the overlap costs, and whether there is a wait window for a metric gate. Canary Blue-green Rolling Traffic pattern Steps through shares you choose (default 5% → 25% → 50% → 100%), each held for a wait window One cutover, 0% → 100%, once the target is healthy Replica by replica, traffic following the replica ratio Extra capacity Near source size; the target grows one step before the source drains that share Both deployments at full size until the source drains Source's replica count at each step; one extra replica mid-step Typical duration One cold start plus a wait per step (at least 390 s each with a metric gate) One cold start plus 30 s propagation; a few minutes One cold start per replica; slowest on large deployments Metric gates Yes, after every step No (no wait window) No How to go back Cancel freezes the current share, then run the rollout in reverse Run the rollout in reverse; --final-source-replicas 1 keeps the old model warm for...