Best Practices For Multi Turn RL

Best Practices for Multi-Turn RL

Join us for our inaugural conference, Forge 2026

Blog

Best Practices For Multi Turn Rl Best Practices for Multi-Turn RL

PUBLISHED 12/10/2025 Impact at a Glance

2X Faster Proposal Assistance

0.75s Sub-second first-token latency

125% Increase in Engagement

Table of Contents Introduction From Single-Turn to Multi-Turn Why RL Beats SFT for Multi-Turn Agents Anatomy of a Multi-Turn RL System Reward Design: Partial vs Trajectory-Level Building a Reward Function that Survives Contact with Reality Practical Recipes: Making Multi-Turn RL Work Case Study: A Deep Research Agent Summary

Table of Contents

How to train LLM agents that can reliably plan, call tools, and recover from their own mistakes. Introduction

In the evolution of AI agents, we are witnessing a distinct phase shift. We are moving past the era of simple, single-turn query-response interactions implemented in simple assistants into a domain defined by sequential agency. We are no longer just asking models to answer a question; we are asking them to interface with an environment through multiple interactions to complete complex tasks that are multi-step, open-ended, and tool-heavy – e.g.,: • Planning a trip: search, filter options, compare itineraries, book flights, reserve hotels, and monitor changes. • Data investigations: run SQL queries, inspect anomalies, call monitoring dashboards, and generate follow-up hypotheses. • Research workflows: search, read, summarize, cross-check, and synthesize across multiple sources.

These are multi-turn, sequential decision problems: the agent must decide which tool to call, when, and with what arguments, potentially dozens of times before the task is complete. The environment state evolves with every tool call. In this regime, Supervised Fine-Tuning (SFT) on “golden traces” is often not enough. We need agents that learn from interaction – from success, failure, and everything in between. That is exactly what Reinforcement Learning (RL) provides. This post lays out how to think about multi-turn RL for sequential tool use, how to design reward functions that don’t explode in your face, and a set of practical recipes that we have found to work in real systems. From Single-Turn to Multi-Turn

Before we talk about training, it’s worth contrasting the single-turn and multi-turn settings. Feature Single Turn Multi Turn Interaction space Interaction space Small, fairly predictable (e.g., “call one API once”) Combinatorial explosion of tool sequences and arguments Credit assignment Trivial: the one call was right or wrong Hard: final failure may come from any earlier decision Training method SFT usually works quite well SFT struggles; RL is typically needed Example “Use search once and answer” “Perform deep research with many calls, cross-checks, and revisions”

A common workaround is to approximate multi-turn tasks by decomposing them into single-turn subproblems – for example, treat each retrieval call as an independent “episode” with its own label (“did we retrieve a relevant document?”). This is viable only if you can define good partial rewards on each step. As tasks become longer-horizon and more open-ended, decomposition breaks down: what matters is whether the final outcome is good, not whether each individual step looks locally reasonable. That’s where full multi-turn RL comes in. Why RL Beats SFT for Multi-Turn Agents

What SFT gives you

Supervised Fine-Tuning takes a dataset of expert trajectories – “golden paths” where a human or stronger model has already solved the task – and trains the model to imitate those traces token-by-token. This has clear benefits: • It jump-starts the agent with reasonable behavior. • It can teach non-obvious tool usage patterns when those examples appear in the data. • It is stable and easy to scale.

But SFT has equally clear limitations in the multi-turn setting: • It sees only a small subset of the state space. Once the agent deviates from the golden path – because the environment changed, a tool failed, or the user asked something novel – it has no guidance. • It cannot learn to recover. Failures in intermediate steps are rarely labeled with what to do next. The model simply never sees those trajectories. • It requires fully labeled trajectories. For complex tool chains, generating high-quality supervised traces is expensive and brittle.

In other words, SFT learns a map of known roads , but not how to navigate when you miss a turn or a road is blocked. What RL adds

RL, in contrast, optimizes the policy to maximize expected cumulative reward in the actual environment: • It encourages exploration, discovering new strategies that were never present in the supervised data. • It learns from interaction rather than from static logs: every rollout is a new training example. • It naturally supports error recovery: if taking a corrective action after a mistake leads to higher final reward, the policy can learn to do that.

Crucially, RL is not purely an alternative to SFT; it is usually stacked on top of an SFT-ed model. SFT gives you a warm start; RL teaches the model to adapt, generalize, and self-correct. In multi-turn, tool-heavy settings, RL is usually the way to go. SFT alone quickly hits a ceiling. Anatomy of a Multi-Turn RL System

A realistic multi-turn RL training loop for LLM agents looks more like a distributed systems diagram than a simple “environment + agent” cartoon. The slide deck shows a helpful schematic: prompts flow into a trajectory generator , which talks to an inference service (the LLM), an environment (the tools), and a trainer , forming a closed loop. Let’s unpack the key components. Key Components Block Diagram Trajectory generator

The trajectory generator: • Takes initial prompts (tasks). • Calls the policy (LLM) via the inference service to obtain the next action (e.g., a tool call). • Executes that action against the environment (APIs, databases, search). • Captures the output of the action (tool call results). • Terminates the episode when some stopping condition is met (success, failure, timeout, max steps).

The output is a trajectory (a rollout): a sequence of user-assistant-tool messages capturing the interactions between different roles in the conversation. Environment and tools

The environment is everything outside the LLM: • Tool backends (search APIs, booking systems, code execution sandboxes, etc.). • State that evolves over time (e.g., a simulated calendar, a running...