Boosting Mtp Acceptance Rates In Baseten Speculation Engine

Open-sourcing Baseten’s suffix automaton MTP accelerator Announcing our Series F . Learn more

Model performance

Open-sourcing Baseten’s suffix automaton MTP accelerator

We've open-sourced a library from the Baseten Speculation Engine to boost MTP acceptance rates by up to 40% using Suffix Automaton Decoding.

Authors

Mahmoud Hassan

Model Performance Team

Last updated January 23, 2026

Share

Link to open-source library here . Two common approaches are widely adopted for speculative decoding: N-gram speculation , which predicts the next token based on fixed-length patterns from recent context.

Draft model speculation, like EAGLE or multi-token prediction (MTP), which uses a smaller, faster neural network to predict several tokens ahead.

At Baseten, we developed a hybrid method to effectively batch the token verification phase, significantly reducing latency for individual requests. Our method combines a suffix automaton—an advanced form of n-gram lookup—with an MTP/EAGLE draft model. This approach is particularly impactful for applications like code generation, where long, repetitive patterns are common. For the open source version, we’ve integrated our technique into NVIDIA TensorRT-LLM so that anyone can use it. We achieved these speedups with zero added overhead, making it suitable for production deployments. On production agentic coding workloads, we see up to 40% higher throughput at equal latency and up to 40% lower latency at equal throughput, compared to MTP alone. ✕ Throughput per single request without speculative decoding, with multi-token prediction (MTP) and with Baseten’s hybrid MTP + suffix automaton (SA) approach. Testing with nvidia/DeepSeek-V3.1-NVFP4 on the dataset glaiveai/code-edit-samples, we see 30%-33% higher acceptance lengths and throughput across different batch sizes using our hybrid approach than MTP alone. ✕ Average acceptance length without speculative decoding, with MTP, and with the MTP/SA hybrid approach. Testing with nvidia/DeepSeek-V3.1-NVFP4 on the dataset glaiveai/code-edit-samples, we see 34% higher acceptance lengths using our hybrid approach than MTP alone. Suffix Automaton Decoding This approach improves upon n-gram lookup decoding by using a suffix automaton (SA) for prediction lookups. Unlike the fixed-size pattern matching available in vLLM and TensorRT-LLM’s n-gram speculative decoding, SA decoding identifies arbitrarily long patterns and selects the longest possible match. Additionally, the suffix automaton is updated in real time during generation, resulting in higher acceptance rates on long sequences.

Combining MTP and SA Decoding SA Decoding shines at code generation, where the accept length is 10+ with long context, but performs poorly on reasoning and other writing tasks, with accept rates near 0. Meanwhile, MTP produces consistent speed-ups across all domains, though the accept rate is usually only 2-4 tokens per iteration. Baseten Speculation Engine, which is a core component of the Baseten Inference Stack , combines both approaches: if the SA finds a match longer than a threshold then the SA match is used, otherwise MTP is used. This achieves a significant speedup on MTP models like  DeepSeek-V3.1-NVFP4 . The level of speedup depends heavily on the task, with a more pronounced increase on agentic coding and math tasks. We commonly see up to 40% improvements on coding applications for production workloads.

Integrating with TensorRT-LLM for the open-source community To integrate with the TRT-LLM runtime, requests are processed as follows: The suffix automaton for the initial prompt is constructed on the host, overlapping with the KV-cache prefill on the device.

Before the first generation step, the automaton state is transferred to the device.

During generation, the suffix automaton is updated directly on the device , without introducing additional synchronization points.

The suffix automaton itself is a highly efficient data structure, with an amortized runtime complexity of O(1) per update. As a result, by carefully scheduling its construction and updates to avoid new synchronization points, near-zero overhead is achieved. ✕ How Baseten’s hybrid MTP + suffix automaton decoding is integrated into TensorRT-LLM across the context and decode loops. During context setup, KV prefill runs on the GPU while the initial suffix automaton state is built on the host, then transferred to the device. In the decode loop, SA states are updated in parallel on the GPU alongside MTP draft sampling and verification, enabling higher throughput with no added synchronization overhead. To support this design, we built a Python API that exposes three core operations (check out the full API in the repository here ): add_request(request_id: int, prompt: list[int]) : builds a suffix automaton state on the host.

prepare(request_ids: list[int]) : prepares a GPU batch, copying newly created suffix automaton states to the device.

extend(draft_tokens_out: tensor, accepted_tokens_in: tensor) : a CUDA-kernel, CUDA-graph-compatible operation that updates a batch of suffix automaton states on the GPU and returns SA draft tokens and match lengths (i.e., confidence scores). A batch of N requests is updated in parallel by launching a grid with one block per batch slot and 1 thread each.

extend is called before draft sampling to compute match lengths from the suffix automaton. These match lengths are compared against a threshold to decide how many draft tokens come from suffix-automaton continuations versus multi-token prediction sampling . To achieve high performance while avoiding platform-specific core logic, suffix automaton states are represented as plain old data (POD) structs, and the core algorithm is implemented in a header-only implementation that compiles for both C++ and CUDA. This allows the suffix automaton logic to be written once and run on both the CPU and GPU. Additionally, the use of POD structs enables efficient, low-overhead data transfer between host and device. This implementation demonstrates the high level of interoperability between C++ and CUDA. For example, we implement a POD graph structure along with a dynamic hash map for storing suffix automaton states, both of which are fully C++ and CUDA-compatible. By embracing POD structs, all we need for CUDA support is a C++ smart pointer type with CUDA specializations for malloc and memcpy . Finally, we achieve torch stream capture (CUDA graph) compatibility by...