Post-Training at the Frontier

RL Post-Training Nemotron 3.5 Lightning | CoreWeave Blog

Announcement

Webinar

Podcast

GTC 2026

CoreWeave Forge is here: connect your AI loop and compound every improvement. Read more

Infrastructure

CoreWeave Forge

Solutions

Resources

Log in

| Sign up Contact us

Contact us

Clear

In early September, CoreWeave, NVIDIA, and You.com ran a three-week sprint to post-train an open-weight model for agentic web search, using web-grounded search data, supervised fine-tuning (SFT), reinforcement learning (RL), and managed  RL Rollouts . The final workflow reached 45.45% accuracy on BrowseComp , compared with 36.97% for the original baseline—a 22.94% relative improvement—with 31.55% fewer tool calls per task. For teams building search agents, that means more questions answered correctly with fewer tool calls. For teams running RL post-training, better training data is only part of the challenge. Inference must generate rollouts, training must update the policy, and the updated weights must reach serving replicas before the next cycle. Conventional redeployments can interrupt rollout generation and leave accelerators waiting. CoreWeave RL Rollouts, using NVIDIA Dynamo and related serving technologies, is designed to reduce that synchronization overhead. This technical deep dive examines You.com ’s data and training strategy, CoreWeave’s managed RL Rollouts infrastructure, NVIDIA’s model and serving technologies, and the results and practical lessons from the collaboration. The bottleneck in RL post-training: Inference cadence Open-weight models have improved rapidly, but foundation models alone are not enough. Teams post-training models for specific tasks—search, reasoning, code generation—now have one constraint: keeping their RL loop productive. In a typical RL loop, the cycle is: generate rollouts → score them → update the policy → repeat. Efficiency during RL Rollouts can be a big bottleneck in the post-training stack. Most infrastructure treats these as separate problems. RL infrastructure requires treating them as one. The three companies addressed complementary parts of the problem: You.com supplied search and web-grounded training data; NVIDIA contributed the Nemotron model, post-training, and serving technologies; and CoreWeave provided the managed training and rollout  infrastructure that connects these stages. The work: Web-grounded data, difficulty filtering, and GRPO You.com started with a problem statement: open datasets age fast and lean heavily on Wikipedia. Real questions live on the real web. So they built training data from the web graph itself. Instead of open-source benchmarks, You.com generated multi-turn QA pairs grounded in objective facts on real web pages. They sampled from the middle popularity band—skipping the spam-heavy tail and boilerplate-heavy head—then followed 1-hop links to pull page content from their index. The result: questions whose answers lived on the web, not in archived datasets. Then came difficulty filtering. You.com ran each question eight times through Nemotron 3.5 Lightning using their own search APIs, scoring each attempt with an LLM judge. They bucketed questions by pass rate: too easy (8/8), easy, medium, hard, too hard (0/8). The hypothesis was simple: if the model sometimes solves it, it's trainable. If it never does, it's not worth the compute budget yet. The team used 4,932 SFT traces generated with You.com Search and  NeMo Automodel , followed by 1,480 RL tasks trained with group relative policy optimization ( GRPO ) and a binary reward signal. The source draft reports ~1 hour for SFT on four NVIDIA HGX B200   GPUs and  ~8 hours for the RL run. The results showed that the infrastructure made the timeline possible, but the data strategy made it count. The infrastructure: RL Rollouts and checkpoint synchronization CoreWeave and NVIDIA focused on the rollout side of the training loop: reducing the time it takes to get each new policy onto inference replicas. In a conventional rolling redeploy, each replica restarts, downloads the full checkpoint from object storage , and waits for the engine to initialize while other replicas keep serving. The restarting replica is unavailable during that time, and its unfinished requests can be lost. Repeated across a training run, those interruptions add up.

Illustration of the collective RL training loop between You.com and CoreWeave RL Rollouts replaces these restarts with in-place weight updates, which we call Hot-Load . The trainer publishes a checkpoint and calls the Hot-Load API while earlier rollouts are still running. CoreWeave's control plane coordinates the update across replicas and tracks the checkpoint version each one serves. NVIDIA Dynamo, NVIDIA ModelExpress , and vLLM handle the update within each replica.  Before loading the new weights, the replica pauses serving but keeps its in-flight requests and cache in memory. Once the weights are loaded into the existing engine, serving resumes and those requests continue on the new weights. New requests queue during the pause. As with a rolling redeploy, other replicas keep serving; the difference is that the updating replica resumes serving much faster and preserves its state. This lets trainers overlap weight updates with rollout generation and allow a request to span multiple policies. For strictly on-policy training, the trainer can request a prompt-cache reset and wait for every replica to finish updating before sending the next batch. Delta weight sync reduces both the amount of data the trainer uploads and the time replicas need to load weights compared with full weight sync. Rather than publishing the full model each time, the trainer publishes a compressed delta against a baseline checkpoint. In our Nemotron 3.5 Lightning tests, these deltas were roughly 0.14%–0.95% of the full BF16 checkpoint size. The weight-distribution design also avoids having every replica download from object storage: one replica reconstructs the checkpoint locally, and the others receive the weights via GPU-to-GPU transfers over CoreWeave’s InfiniBand network or NVIDIA NVLink interconnect. In a benchmark using Nemotron 3.5 Lightning with four replicas on CoreWeave’s NVIDIA GB300 NVL72 systems , the fastest delta Hot-Load was roughly 15× faster than the average cold-start redeploy. For training workflows that repeatedly refresh inference weights, faster updates can reduce waiting between rounds and keep GPUs doing useful work for...