Dflash Faster Llm Inference

DFlash: 3x faster LLM inference Announcing our Series F . Learn more

Model performance

DFlash: 3x faster LLM inference

DFlash beats EAGLE's ~2x speedup ceiling with parallel token prediction. Baseten's implementation hits 3x speedups on Qwen3-8B, 10-30% faster than vLLM.

Authors

Aaryam Sharma

Last updated May 8, 2026

Share

By design, LLMs today only output one token at a time. As a result, speculative decoding (SpecDec) has become one of the most effective ways to improve LLM inference latency: a smaller draft model proposes tokens that the target model verifies in parallel, yielding 2x+ speed improvements depending on proposal acceptance rates. EAGLE ( original , EAGLE-2 , and now EAGLE-3 ) emerged as one of the most popular SpecDec techniques, using the target model's hidden states to predict draft tokens with better acceptance rates than earlier approaches. But EAGLE is autoregressive; each predicted token requires its own forward pass from the draft model, which in practice often caps its speedups at roughly 2x as it lets errors accumulate across forward passes. DFlash was introduced in February 2026 ( paper , code ) to push past the ceiling that autoregressive drafting imposes on SpecDec speedups. A single DFlash forward pass completes faster than the entire EAGLE draft phase, while predicting several more tokens. In practice, we see that translates to a 50% speed bump: on Qwen3-8B on a single B200, Baseten's DFlash implementation delivers ~3x speedups across various benchmarks (in terms of inference latency and throughput), compared to 2x with EAGLE. ✕ Throughput for 3 different speculative decoding techniques (and one baseline without SpecDec) on the GSM8k benchmark. Baseten’s DFlash implementation achieves the highest mean throughput at 654 TPS, a 3x improvement compared to baseline and 10% (1.1x) faster than vLLM’s DFlash implementation. ✕ Latency for 3 different speculative decoding techniques (and one baseline without SpecDec) on the GSM8k benchmark. Baseten’s DFlash implementation achieves the lowest mean latency at 1.2 seconds, a 2.9x improvement compared to baseline and 25% (1.3x) faster than vLLM. This post covers how DFlash works, how Baseten's implementation differs from vLLM and SGLang, training details, and full benchmark results, including inference latency and throughput across GSM8k, MATH-500, and NVIDIA’s Nemotron post-training dataset. DFlash: Block diffusion for speculative decoding DFlash aims to bridge the quality of autoregressive decoding with the speed of diffusion LLMs. Like EAGLE, DFlash uses the hidden states of the target model as input features, as they provide very rich data representations aligned with the target model. But unlike EAGLE, DFlash: Predicts multiple (γ) tokens in parallel in a single forward pass by using bidirectional attention

Allows for a much deeper draft model without sacrificing speed

Although a single forward pass for the larger DFlash draft model might be 2-4x slower than a single forward pass for EAGLE (depending on model sizes), DFlash predicts 8-16 tokens at once, while EAGLE could only predict 1. That means that despite the larger draft model, a single DFlash forward pass is faster than the entire EAGLE draft phase, while also predicting several more tokens. The larger model size also allows DFlash draft models to learn more and produce higher-quality drafts. The DFlash model looks like this: ✕ The DFlash draft model takes fused target context features (hidden states from the target model), the last valid decode token, and a block of mask tokens as input. It then runs these through multiple draft layers using bidirectional attention — predicting all masked tokens in parallel in a single forward pass — before passing them to the target language model head for speculative verification. DFlash in the Baseten Inference Stack For drafting the new tokens, DFlash creates a block of size block_size (a parameter), with the first token being the last valid token (either the last context token or the last accepted token), and the rest being mask tokens. Then, similar to masked language modeling, the DFlash model proceeds to unmask the mask tokens. In practice, we take the hidden states from 5-6 evenly spaced layers of the target model. The same hidden states are passed through all layers of the DFlash model. These are projected to construct the KV cache for each layer. We have failed to find a faster DFlash implementation than our own. To make DFlash so fast, we tested each backend to leverage the fastest one and implemented a custom forward pass mechanism to prepare the KV cache and produce output tokens. We also support guided decoding. Results: DFlash speed improvements on 3 benchmarks To assess inference latency and throughput improvements from using DFlash, we tested our implementation with Qwen3-8B across three benchmarks. We used GSM8k and MATH-500 because they were used in the original DFlash paper (making the results directly comparable), and we added a diverse prompt dataset (NVIDIA’s Nemotron post-training dataset covering science, coding, and general chat) to test generalization beyond math. Our DFlash implementation outperforms both EAGLE and other DFlash backends (vLLM, SGLang) across all three benchmarks. We don't report the numbers for SGLang here since we could not reproduce reliable results (more details below), but even after cleaning up the output sequences post-hoc, both our imlementation and vLLM were faster that SGLang. GSM8k (arithmetic reasoning) The GSM8k benchmarks were conducted at concurrency 16 on a single B200 GPU. Throughput results (higher is better):

And latency results (lower is better):

SGLang also supports DFlash, however, in our evaluation, SGLang produced unreliable results due to a high rate of looping outputs. In these cases, a sequence of tokens near the end of the response was repeated until generation completed. Because these repeated tokens did not represent meaningful generation, the resulting outputs were unsuitable for evaluation and resulted in unusually high acceptance rates and throughput. This made the aggregate statistics unrepresentative of usable generation quality. Although we filtered out these samples, this altered the output length distribution and biased our data towards shorter and faster generations. We therefore decided to exclude SGLang from our comparisons as these were not directly comparable. In the filtered results,...