You Need A Verifier

You don’t need a frontier model. You need a verifier.

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Back to Blog

-->

In brief

On agentic search, cheap models usually produce a correct answer and then fail to pick it. Majority voting cannot fix that, because it cannot overturn a popular wrong answer. Adding an independent verifier that re-researches each candidate and vetoes the wrong ones beats published SOTA on FACTS-Search, and once you train your own 8B verifier it costs almost nothing, works with any generator pool, and generalizes to a new benchmark and retrieval tool.

Figure 1: The pool is fixed and only the verifier changes. Majority voting alone scores 83.3, below published SOTA. An untrained Qwen3-8B verifier reaches 84.8, still short of it. A Claude Opus verifier adds 10.1 points and $2.94; our trained 8B verifier reaches 92.9 for one cent more than no verifier at all.

The default way to make an agentic system more accurate is to buy a better generator – a frontier model. It works, but it is the most expensive option available.

This post is making a different argument: how you run your models matters more than which model you run. The system below adds an important component to any AI architecture: a verifier. This component increases the performance of every architecture, whether it’s an ensemble of frontier models or cheap open source models.

Why agentic search needs a verifier

Your cheap agent probably already found the right answer. The problem is that it didn’t pick it.

Model runs are not deterministic. The same agent, same question, fails on one attempt and succeeds on the next. Across every generator we tested, pass@k sits far above single-shot accuracy (pass@1): correct answers are routinely present in the pool and simply not selected. So the binding constraint on agentic search is selection, not generation.

Selection is tractable because checking is easier than generating (at least for agentic search QA). Answering “Which band played ‘Johnny Come Lately’ on the Steve Earle album that includes ‘Little Sister’?” takes a chain of searches, dead ends, and synthesis. Verifying a proposed answer is far narrower: research one claim and rule on it.

So why not just take the majority vote when using an ensemble? Because voting cannot overturn a popular wrong answer. When half the pool confidently repeats the same wrong entity, voting does not filter the error, it certifies it. Cheap pools fail this way most, because correct answers are less frequent.

Figure 2: The three-stage loop on a FACTS-Search question. Generators produce k candidates, the verifier researches each one independently and returns a verdict, and the aggregator votes among verified answers only. Here the plurality answer survives; the value of the loop is the cases where it does not.

Verification works

First, ignore cost and run the best verifier available.

We work on FACTS-Search , Google DeepMind’s benchmark of hard multi-hop factual questions that require web search to answer. Every model in this post, generator or verifier, uses the same Brave Search API, so measured differences are model skill and not tooling.

Generators ensemble of closed-source and open-source models (Claude Haiku, Sonnet, Opus, Qwen3-Coder-30B-A3B at k=4) scores 83.3 under plain majority voting and 93.4 with a Claude Opus verifier on top , past the published state of the art of 89.4 ( GPT-5.6 Sol ), as can be seen in Figure 1.

The headroom is larger at the cheap end

Frontier-on-frontier proves the mechanism. The regime that matters in practice is cheap generators, and there the headroom grows rather than shrinks: a noisier pool still surfaces correct answers, they are just outnumbered, which is exactly when identifying them pays.

Our all-open-source pool (Qwen3-14B and Qwen3-Coder-30B-A3B at k=4) votes to 60.1. Put a Claude Opus verifier on those same candidates and it scores 80.4 at $1.76 per question . Twenty points already sitting in the pool, waiting for a verifier good enough to claim them.

Figure 3: The same experiment on an all-open-source pool. Majority voting scores 60.1 and an untrained Qwen3-8B verifier adds 2.4 points. A Claude Opus verifier lifts the pool 20.3 points at 105x the cost of our trained 8B verifier, which recovers 16.9 of those points for $0.017 per question.

The verifier is the expensive part, so we trained our own

A frontier verifier runs a full web-research loop on every candidate: k samples times the number of generators, sixteen loops per question in our best configuration. That costs more than generating everything it checks: the 93.4 result runs $4.26 per question, most of it the verifier.

The obvious fix is a small open-source verifier, and it does not work. A stock Qwen3-8B verifier moves the cheap pool from 60.1 to 62.5. Plus 2.4 points against Opus’s plus 20.3 on the same candidates. That 18-point gap is the training problem: the verification skill does not come free with the parameters.

Training

Given a question and a candidate answer, the verifier runs its own Brave-search ReAct loop and emits one verdict. The reward is a binary exact match against the ground-truth label, so it is fully verifiable, no reward model. Training data is about 6K (question, answer, VALID/NOT_VALID) triples from generator rollouts, deduplicated, class-balanced, and split at the question level.

The recipe is SFT then RL, for the same reason as DeepSeek-R1: RL from a cold start has no competent policy to explore from. Two failure modes here are specific to verification. RL alone reward-hacks the class imbalance, with tool-use rate falling from 94% to 0% in nine steps. Balancing the data closes that shortcut but not the underlying problem: the policy settles into one narrow search behavior and cannot climb out of it, so search quality plateaus wherever the cold start happens to land. SFT alone teaches the search skill and dilutes the verdict, because imitation spreads its loss over every token of a long trajectory while the verification signal lives in one token. Composed, tool use stays high and becomes adaptive, and the verdict sharpens without the policy collapsing.

Measuring any of this needs two things. Re-running the full loop for every checkpoint is too slow, so verifier variants are scored offline against a fixed pool of pre-generated rollouts: the candidates never change,...