Mercury 2 For Search

Mercury 2 for Search: Fast enough to run a hundred times per query – Inception

10x free tokens on Mercury 2

|

Claim yours

10x free tokens on Mercury 2. Claim yours

Blog

/

Product

Mercury 2 for Search: Fast enough to run a hundred times per query

Apoorv Saxena

For the past two years, “search” stopped meaning a ranked list of links and started meaning an agentic pipeline. Classify, rewrite, fan out, rerank, and synthesize. Fifty to a hundred LLM calls per search query, almost all sequential. That's a brutal place to put an autoregressive model: a 400ms rewrite blocks retrieval, which blocks reranking, which blocks the first word the user sees. So teams cut the pipeline down until it's shallow enough to be fast. The industry has split into two camps: synchronous search running thin, cheap models that barely think, and deep-research agents that take thirty minutes. Nobody ships the thing in the middle: a pipeline deep enough to be smart and fast enough to be useful. Mercury 2 decodes over 1000 tokens per second on standard NVIDIA GPUs. Fast enough to run every step inside the latency budget you already have. Latency is the quality budget In most products, latency and quality are separate dials. You can make the answer better by letting the user wait. In search they're the same dial, because every step that improves makes the answer is itself an LLM call that increase latency: Query rewriting finds the documents a literal match misses.

Reranking narrows the merged set to the passages worth synthesizing from.

Snippet summarization keeps the synthesis context clean enough to cite.

Each step is an LLM call. Every one of them is either blocking the user or generates output the user reads. Agents that run six rewrites instead of one, rerank a hundred results instead of ten, and summarize every retrieved page don't just retrieve faster, they retrieve better. Why diffusion doesn't need to go one token at a time Autoregressive models generate left to right, one token per forward pass. Every token waits for every prior token. That serialization is the structural bottleneck, and it's why the industry's answer to search latency has been "use a smaller model". That’s the only lever available when the decode order is fixed. Diffusion language models don't have a fixed decode order. Mercury 2 generates in parallel, refining a whole span of output across a small, bounded number of steps rather than emitting one token per pass. Each step is a full-context forward pass over the whole span. The model sees the developing answer holistically and converges on the final output over successive passes. A 300-token reranking rationale and a 500-token snippet summary come back in a fraction of the time. A search pipeline generates hundreds of them. The fastest model in every step, measured "Fast" in a search pipeline isn't one number. What matters is how long each step takes, because each step blocks the next one. So we measured per-step latency directly on WideSearch , a benchmark of exhaustive info-gathering tasks requiring dozens of live queries each. Same agentic harness, same 100 tasks, same live retrieval, four models, every LLM call timed. Reproducibility: WideSearch benchmark, 100 English tasks, live Exa retrieval. Latency medians over 336–493 calls per model, each on its own provider's public API. Harness and per-call timing are public at: github.com/apoorvumang/retrieval-vs-recall

Mercury 2 is the fastest model at every step of the pipeline, almost 2x faster than Gemini 3.1 Flash Lite on query planning, 4.7x faster than Claude Haiku 4.5, 10x faster than GPT-5 Mini. That per-step gap is where the end-to-end numbers in the next section come from: a pipeline is only as fast as the sum of its hops. Doing the math on a real pipeline Take a mid-complexity answer-engine query with a 2-second budget to first token. A conventional pipeline spends that budget on one rewrite, a single retrieval pass, no LLM rerank, and a synthesis step that starts streaming late. Most of the 2 seconds is decode time, and the quality ceiling is set by whatever the retriever happened to return. The same budget on Mercury 2 buys four parallel rewrites, retrieval fan-out across all of them, an LLM rerank over the merged candidate set, per-document snippet summarization, and a synthesis step that starts streaming with room to spare. Same latency envelope. A structurally better answer, because the pipeline did the work. At $0.25/M input and $0.75/M output, that deeper pipeline also costs less per query than the thin one running on a GPT-5 Mini or Haiku 4.5. Quality at full speed Speed only matters if the answers hold up. So we ran the cost-efficiency generation tier on two grounded-search benchmarks: FRAMES (multi-hop retrieval and synthesis, 2–15 Wikipedia articles per question) and DeepSearchQA (open-ended agentic search, graded on whether the answer set is complete). n=100 per benchmark, identical question set for every model, all at medium reasoning effort. Agentic tool-calling loop over Exa; 25-call budget on FRAMES, 30 on DSQA. Graded with each paper's official prompt and mandated judge — GPT-5.4 for FRAMES, Gemini-2.5-flash for DSQA. Cost includes Exa retrieval at list rates and bills input at full price with no prompt-cache discount. *Claude Haiku 4.5 routed via OpenRouter (direct key depleted); its latency includes a proxy hop and is likely a couple of seconds high.

On FRAMES, four models land within three points of each other — 0.78, 0.78, 0.78, 0.81. At n=100 the standard error is four to five points, so that spread is noise. On multi-hop grounded QA this tier is tied, and GPT-5 Mini's nominal top score isn't a real lead. DeepSearchQA separates them, and not in Mercury 2’s favor. GPT-5 Mini scores 0.44 against Mercury 2’s 0.34 — a ten-point gap, wider than the spread among the other three, though still inside two standard errors at this sample size. DSQA rewards breadth: finding every item that matches a constraint, rather than reasoning through a chain. A model that plans decisively and reads widely does better there, and GPT-5 Mini does. What doesn't change on either benchmark is what a right answer costs and how long you wait for it. Mercury 2 finishes a FRAMES query in 10.8 seconds, against 19.8 for Gemini 3.5 Flash Lite, 21.0 for Claude Haiku 4.5, and 38.8 for GPT-5 Mini. And per correct answer (the number that reaches your bill) Mercury 2 is the lowest of the four on both benchmarks: $0.047 on...