How We Built The Worlds Fastest Api For Glm 52

How we built the world’s fastest API for GLM-5.2 Announcing our Series F . Learn more

Model performance

How we built the world’s fastest API for GLM-5.2

GLM-5.2 at 280+ tokens per second on NVIDIA Blackwell with KV-aware routing, PD disaggregation, Multi-Token Prediction, NVFP4, and more optimizations

Authors

Alex Korte

Magdy Saleh

Tri Dao

Anant Desai

Bryce Dubayah

Abu Qader

Philip Kiely

Last updated June 23, 2026

Share

TL;DR We built the world’s fastest API for GLM-5.2, achieving over 280 TPS as measured by Artificial Analysis. This leading performance is thanks to a suite of performance optimizations across the Baseten Inference Stack, including our optimized runtime engine, NVFP4 on NVIDIA Blackwell, KV-aware routing and PD disaggregation with NVIDIA Dynamo, and Multi-Token Prediction heads for speculation.

GLM-5.2 is the biggest news in open models since DeepSeek-R1. ✕ Tech leaders are embracing GLM-5.2 as a new high water mark of frontier intelligence on open models It’s easy to see why. GLM-5.2 delivers comparable performance to GPT 5.5 and Opus 4.8 at a fraction of the cost, generally 70-80% less expensive on a pure token basis (use our calculator to estimate savings for your workload ). But a model has to be more than just smart and inexpensive. To be useful in production, a model needs to be fast, reliable, and available at scale. Delivering on the promise of frontier open intelligence requires exceptional inference. Accordingly, we built the world’s fastest API for GLM-5.2, currently serving over 280 tokens per second as measured by Artificial Analysis. ✕ GLM-5.2 runs at SOTA speeds on Baseten model APIs, measured by Artificial Analysis June 22, 2026 We achieved this performance by leveraging a number of techniques across the entire inference process by: Updating our custom inference engine to implement shared DSA for the GLM-5.2 architecture.

Running and calibrating an in-house NVFP4 quantization from the original FP8 weights that demonstrates equivalent quality on agentic benchmarks like BFCL.

Ensuring high KV cache hit rates via KV-aware routing built with NVIDIA Dynamo tools for lower prefill burden and improved TTFT on requests with repeated prefixes.

Achieving a 2x higher TPS for observed workload shapes by running disaggregated inference built with the NVIDIA Dynamo toolkit.

Improving TPS further via speculation by implementing support for GLM-5.2 Multi-Token Prediction heads.

You can experience this performance for yourself with GLM-5.2 on Baseten Model APIs . We also have GLM-5.2 available as a dedicated deployment for high-volume workloads. ✕ Notion offers GLM-5.2 via Baseten GLM-5.2 Overview GLM-5.2 by Z.ai is a 744B parameter frontier LLM that excels at agentic tasks (especially coding) and supports up to a 1 million token context window. It uses a similar architecture to its predecessor, GLM-5.1: mixture of experts (40B active parameters), non-thinking and thinking modes, and a fully open MIT license. While GLM-5.2 shares a lot in common with GLM-5.1, it now uses shared DSA weights, which we implemented support for in our customized runtime engine. ✕ GLM-5.2 offers frontier performance across tasks. Image from Z AI. GLM-5.2 has great benchmark scores, but by now AI builders know that there is more to a model’s utility than its performance on standard evals. In practice, GLM-5.2 meets or exceeds the capabilities suggested by its benchmarks. It's a genuinely great model for writing code, operating agents, and other frontier language model tasks. High-quality NVFP4 quantization for Blackwell GPUs We run our model APIs on NVIDIA Blackwell GPUs with a customized inference engine within the Baseten Inference Stack . The selected runtime uses NVFP4 weights for maximum performance. From the original FP8 weights, we performed an in-house quantization to NVFP4 using NVIDIA ModelOpt. NVFP4 is a 4-bit floating point data format by NVIDIA that uses dual scale factors to retain high dynamic range and preserve model quality. In our calibration and testing of the quantized model, we focused on ensuring that GLM-5.2 performs faithfully on common patterns for agents. On the BFCL function calling benchmark, we observed roughly equivalent performance between the native FP8 weights and our NVFP4 quantization, with scores across runs within the margin of error for the benchmark. NVFP4 quantization improves performance on both time to first token and tokens per second by unlocking faster tensor cores and reducing burden on VRAM bandwidth. ✕ Performance is excellent across both TTFT and TPS, measured by Artificial Analysis June 22, 2026 Cache-aware routing with NVIDIA Dynamo GLM-5.2 is particularly well suited for long context requests and complex agentic tasks. These workloads generally have very long input sequences. By re-using KV cache between requests, we can skip expensive prefill for shared sequences. We generally talk about KV cache re-use in the context of time to first token (TTFT). However, reasoning models like GLM-5.2 generally care more about time to first answer token (TTFAT), which combines TTFT with some TPS for the reasoning sequence. ✕ Time to First Answer Token for GLM-5.2, measured by Artificial Analysis June 22, 2026 This chart shows that of the 7.9 second average to generate the first answer token, 7.1 of those seconds were spent generating reasoning tokens versus only 0.8 seconds spent processing the input sequence. Still, bringing the TTFT down to 800 ms is important for the overall responsiveness and throughput of the system. In large-scale production deployments, KV cache is split across various independent replicas. We use tools from NVIDIA Dynamo to route incoming requests. ✕ KV-aware routing sends requests to replicas that already have relevant context cached, saving time by avoiding redundant prefill computation. Exact cache hit rates on a multi-tenant API depend on the exact traffic profile at any given time. Thus far, we’re observing high hit rates across fairly heterogeneous traffic, which reduces load on prefill and improves end-to-end performance. Prefill-decode disaggregation with NVIDIA Dynamo One of the highest-impact optimizations we made to our performance is disaggregating prefill and decode for GLM-5.2. There are two distinct phases of LLM inference: Prefill: The compute-bound process that processes the input sequence, builds the KV cache, and generates the first output token. Prefill performance determines...