Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

⚡️ FlashAttention-4: up to 1.3× faster than cuDNN on NVIDIA Blackwell →

Introducing Together AI's new look →

🔎 ATLAS: runtime-learning accelerators delivering up to 4x faster LLM inference →

⚡ Together GPU Clusters: self-service NVIDIA GPUs, now generally available →

📦 Batch Inference API: Process billions of tokens at 50% lower cost for most models →

🪛 Fine-Tuning Platform Upgrades: Larger Models, Longer Contexts →

All blog posts

Research

Published 12/3/2025

Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

Authors

Roman Garipov, Fedor Velikonivtsev, Ivan Ermakov, Ruslan Svirschevski, Vage Egiazarian, Max Ryabinin

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Summary

We introduce AutoJudge, a method that accelerates large language model (LLM) inference through task-specific lossy speculative decoding. Instead of matching the target model’s output distribution token by token, this method identifies which specific generated tokens affect downstream quality. Compared to prior approaches, AutoJudge does not require manual annotation, as it employs a classifier trained in a self-supervised manner.

AutoJudge can accept up to 40 draft tokens per verification cycle with only a slight accuracy drop, achieving 1.5–2x speedups over standard speculative decoding, and is easy to integrate into existing LLM inference frameworks. We will be presenting our research findings on AutoJudge at NeurIPS 2025 — come meet the team to learn more!

Speculative decoding speeds up generation by pairing a small “draft” model with the main “target” model. The draft proposes several next tokens; the target runs in parallel to verify them. Tokens that match the target are accepted; the first mismatch (and everything after) is rejected. This keeps the output distribution identical to the target model’s own decoding. In practice, strict distribution matching isn’t always necessary. Lossy variants trade a tiny amount of quality for more speed. Judge Decoding is one such approach: it only rejects a mismatch if accepting it would harm the final answer. For example, math errors or logical bugs matter, but minor stylistic changes often don’t. Our work builds directly on this idea. The catch with Judge Decoding is data : it needs humans to label “critical” mismatching tokens for each task, which is expensive and doesn’t transfer perfectly across domains. AutoJudge removes this bottleneck by automatically mining those important tokens—no human annotators required. The AutoJudge Method

Figure 1. The data collection stage of AutoJudge AutoJudge consists of the following stages: Automatically mine “important” mismatches For a prompt, generate a target answer and locate where draft and target models disagree. Iteratively swap draft ↔ target tokens and re-evaluate the task (e.g. GSM8K answer equality or code unit tests). A mismatch is important if keeping the draft token breaks the final answer; otherwise it’s unimportant . This semi-greedy pass reliably surfaces at least one important token whenever answers differ. Train a tiny classifier on existing embeddings We use a simple logistic regression fed by transformer hidden states already computed during speculative decoding. Concatenating draft and target token embeddings works best and remains robust across regularization choices and small architectural variants. ‍ Accept “unimportant” mismatches at verification time During the verification phase—exactly where the baseline would reject a mismatching draft token—we call the classifier. If it predicts that the token is unimportant , we accept it and keep moving forward, increasing accepted tokens per speculation cycle. The approach is compatible with standard, tree-based, and single-model multi-head speculative decoding methods, and slots into popular stacks like vLLM, TensorRT-LLM, and TGI. In practice, we pick a high-recall threshold (≥90%) to safeguard accuracy while still skipping a large fraction of tokens.

Figure 2. Example of extra accepted tokens resulting in faster inference In Figure 2, we demonstrate how AutoJudge can accept more tokens during the speculative decoding step. AutoJudge adds a tiny “judge” that asks, at each mismatch, whether the difference actually changes the final answer. In the example, the mismatch is a harmless wording — like “equals” vs “becomes” — and we accept it and keep the rest of the drafted tokens. If it would change correctness — like “+” vs “−” in a math step — we reject it. By only rejecting critical mismatches, we keep longer chunks from the draft, so more tokens are accepted at once and generation is faster with little impact on quality. Performance benchmarks Accuracy-acceptance tradeoffs

Figure 3: Accuracy and the number of accepted tokens on GSM8K for (left) 8-shot Llama-3.2 1B draft / Llama-3.1 8B target and (right) 0-shot Llama-3.1 8B draft / Llama-3.1 70B target (all Instruct). In Figure 3 we show how AutoJudge shifts the speed–quality frontier: as the number of accepted draft tokens per cycle increases (x-axis), AutoJudge (red) stays near the accuracy of lossless speculative decoding while accepting more draft tokens, unlike a naive Top-K baseline whose accuracy drops quickly. This holds in both model pairs (1B/8B on the left, 8B/70B on the right), so you can choose a threshold that yields higher tokens/s with minimal accuracy cost. In Figure 3 (right), we show that it is possible to safely accept three times more tokens, paying only a 1% accuracy drop, demonstrating that speculative decoding can safely accept up to 45 tokens with minimal loss in quality. Inference speedup We integrated AutoJudge into vLLM ’s speculative decoding and measured end-to-end tokens/s on GPUs. (Setups included A100/H100; see notes below for model pairs.) Mathematical reasoning (GSM8K) Across model pairs, AutoJudge delivers consistent throughput gains with small accuracy trade-offs: Llama-3.1-405B (target) / 8B (draft), 8xH100: 91.5% (≈4% drop), 60.1 tokens/s , 1.20× . Llama-3.1-70B (target) / 8B (draft), 4xA100: 89.9% (≈2% drop), 107.4 tokens/s , 1.49× . Llama-3.1-8B (target) / 1B (draft), 1xA100: 80.2% (≈3% drop), 169.2 tokens/s , 1.14× . Baselines: 50.0 (405B), 72.3 (70B), 147.7 (8B) tokens/s.

Programming (LiveCodeBench) AutoJudge automatically identifies…