Frontier Rl Is Cheaper Than You Think

Frontier RL Is Cheaper Than You Think

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Frontier RL Is Cheaper Than You Think Frontier RL Is Cheaper Than You Think The conventional wisdom on RL infrastructure is wrong. Cross-region rollouts, compact deltas, and hot-load updates let teams use distributed capacity effectively instead of waiting for one mega cluster.

PUBLISHED 3/23/2026

CROSS-REGION RL WEIGHT UPDATE LOOP POLICY TRAINER -> SHARED CHECKPOINTS -> GLOBAL ROLLOUT FLEET policy trainer forward/backward + optim step TRAINER OUTPUT base checkpoint every N steps example here: N = 25 changed weights per step: ~2% active checkpoint: policy-model-step-0001-full weight update handoff full checkpoints + compact deltas ACTIVE TRANSFER step 1 checkpoint type: full checkpoint 100.0% of full weights each delta is ~98.0% smaller than a full copy 1024.0 GiB transferred US OHIO ROLLOUT 100% full weight refresh +43 ms inter-region latency +43 ms US VIRGINIA ROLLOUT 100% full weight refresh +58 ms inter-region latency +58 ms EU FRANKFURT ROLLOUT 100% full weight refresh +145 ms inter-region latency +145 ms >98% less traffic typical delta ~2% 50-STEP SAMPLE WINDOW full checkpoint every N steps, example shown here uses N = 25 1 full 100% 5 10 15 20 25 30 35 40 45 50 full checkpoint resets the chain typical delta: about 2% of full each delta is more than 98% smaller than a full copy

On this page

Frontier RL Is Cheaper Than You Think

The conventional wisdom on RL infrastructure is wrong, and it is costing teams that could be competing at the frontier.

The entire mega-cluster narrative rests on a single assumption: that you have to ship 1 TB of weights every time you update your rollout fleet. You do not.

Researchers have spent the last year writing about asynchronous RL and rollout-training disaggregation in systems like AReaL . Teams like Kimi and MiniMax have also published engineering notes on RL parameter updates and asynchronous scheduling. We have been running that pattern in production.

That mega-cluster instinct comes from pretraining, where the main systems problem is keeping one huge synchronous training job saturated. RL is a different problem. The question is not just how to run the trainer. It is also how to keep a large rollout fleet generating data from a fresh enough policy without constantly stalling on full checkpoint transfers.

RL infrastructure in 30 seconds

An RL training run has two jobs:

The trainer does forward pass, reward computation, backward pass, and parameter updates.

The rollout fleet samples trajectories from the current policy, i.e. runs inference on the latest updated model.

The trainer needs dense, tightly coupled hardware. The rollout fleet needs inference throughput across many parallel requests. Pretraining only has the first job. RL has both, which is why the infrastructure question is different.

The 1 TB problem

A typical frontier checkpoint is around 1 TB. If every policy refresh required shipping that full checkpoint to the rollout fleet, then the natural conclusion would be that RL needs one giant co-located cluster with RDMA-class internal networking. Keep trainer and inference on the same fabric, avoid long-distance transfers, and treat remote capacity as second class.

That is the mega-cluster story. It makes frontier RL look like a market only a handful of companies can enter, because everyone else gets boxed out by infrastructure economics before they even get to compete on algorithms or product execution.

But the premise is wrong. You do not need to move the full 1 TB on every update.

The key insight: exploiting 98% sparsity

Between nearby RL checkpoints, most weights change only a little. That makes it practical to send a compressed delta against the previous checkpoint instead of sending the full 1 TB again.

Last year, we empirically observed that more than 98% of weights in bf16 format remain bit-equivalent between consecutive checkpoints, and the unchanged fraction is even higher at lower precision. Our intuition was that post-training updates are extremely fine-grained and RL provides very sparse information signal with just a few bits per rollout. In practice that means RL training uses a fairly small learning rate, and most parameters move only slightly in fp32. Those changes often do not cross the threshold required to alter their 16-bit or lower-precision representation. A recently published paper, Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL , provides a theoretical foundation for the same phenomenon and reports similarly high sparsity, often around 99% in practical RL settings.

In the sample setup behind this post, a full checkpoint is 1024 GiB. The average delta between adjacent checkpoints is 20.3 GiB, or 1.98% of the full model. Over the 50-step window shown below, that cuts cross-region transfer volume by about 94% compared to moving the full model every time.

FULL CHECKPOINT EVERY N STEPS EXAMPLE SHOWN HERE USES N = 25 50-STEP SAMPLE WINDOW grey bars show the cost of shipping a full checkpoint every step 1 5 10 15 20 25 30 35 40 45 50 100% active step ships a full checkpoint 100.00% of full weights 1024.0 GiB transferred WHAT THE DATA GENERATOR IS MODELING seeded sample for this post typical delta: about 2% of full each delta is more than 98% smaller than a full copy full = orange delta = purple Checkpoint Cadence This visual shows the intended RL update rhythm: a periodic full checkpoint, then delta-compressed weight updates in between, so the rollout fleet usually receives only about 2% of the full model. The cadence looks like this: publish a full base checkpoint every N steps, then ship compressed deltas in between. The compression focuses on sending only the changed weights, with checksummed reconstruction so every rollout cluster can rebuild the exact checkpoint losslessly from shared storage.

DELTA-COMPRESSED WEIGHT UPDATES CHANGED WEIGHTS -> COMPACT PAYLOAD -> EXACT RECONSTRUCTION 1. IDENTIFY CHANGED WEIGHTS adjacent checkpoints differ in a few chunks, not everywhere PREV CURRENT DELTA SIZE embed_tokens 00 attn.q_proj 0.2% attn.o_proj 0.4% mlp.gate_up 0.7% mlp.down_proj 0.2% final_norm 0.1% Unchanged chunks stay out of the transfer. Only sparse changed-weight slices survive into the payload. 2. PACKAGE CHANGED TENSORS keep the changed pieces, bit-pack them, then attach reconstruction...