Kimi K2 Thinking At 140 Tps On Nvidia Blackwell

Kimi K2 Thinking at 140+ TPS on NVIDIA Blackwell Announcing our Series F . Learn more

Model performance

Kimi K2 Thinking at 140+ TPS on NVIDIA Blackwell

Our model performance team built the fastest API (300 ms TTFT, 140+ TPS) for Kimi K2 Thinking, a 1T param open model that matches GPT-5 and Sonnet 4.5.

Authors

Abu Qader

Tri Dao

Philip Kiely

Last updated November 12, 2025

Share

TL;DR We launched our Model API for Kimi K2 Thinking and are the best-performing provider on Artificial Analysis with a 300 millisecond time to first token and 140+ tokens per second. This blog post details the model performance work we did to get there. Using the Baseten Inference Stack, we run Kimi K2 Thinking on a single 8xB200 node in NVFP4 with a mixture of Tensor Parallelism and Expert Parallelism plus KV-aware routing for high hit rates on KV cache re-use.

Today, we launched a Model API for Kimi K2 Thinking and achieved state of the art latency as measured by Artificial Analysis . ✕ Baseten runs Kimi K2 Thinking with the lowest latency per independent benchmarks from Artificial Analysis Kimi K2 Thinking is the smartest open-source model ever. With benchmark scores competitive with models like GPT-5 and Claude Sonnet 4.5, the new Kimi model is useful for everything from agents to coding to creative writing. ✕ Kimi K2 Thinking eliminated the intelligence gap between closed and open-source models (image from Artificial Analysis). While it’s just as smart as leading closed models, Kimi K2 Thinking is faster and cheaper. With the Baseten Inference Stack , Kimi K2 Thinking is faster than ChatGPT on GPT-5 with Thinking end-to-end on the same prompt.

Building this high-performance inference service for Kimi K2 Thinking was not easy. At one trillion parameters, Kimi K2 is the largest family of open-source models. Fortunately, Kimi K2 Thinking has a similar architecture to other Kimi K2 models, which is a variant of DeepSeek V3. Accordingly, we were able to re-use a good deal of the performance work we did for the predecessor Kimi and DeepSeek APIs. To run build a high-performance API for Kimi K2 Thinking, we used our Baseten Inference Stack to provide: Robust support for inference on NVIDIA Blackwell B200 GPUs

NVFP4 quantization for model weights (versus original INT4 quantization)

A low-latency parallelism configuration within our inference engine

KV cache-aware request routing for high KV cache re-use rates

This post details the technical work required to get these running together in production. NVIDIA Blackwell compatibility with INT4 to NVFP4 conversion To achieve speeds of over 100 tokens per second, we need to use the most powerful GPU architecture available: NVIDIA Blackwell. With eight B200 GPUs, we can comfortably fit the trillion-parameter Kimi model in a four-bit number format with plenty of memory left over for the KV cache. However, Kimi K2 Thinking is optimized for previous-generation Hopper GPUs. Blackwell GPUs are not available in China due to export restrictions, so labs like Moonshot AI (the makers of Kimi) target Hopper instead. This is most evident in the model’s native INT4 quantization. Ordinarily, INT4 is not suitable for production use. However, Kimi K2 Thinking was created using quantization-aware training during the post training phase, which computes the scales and trains the weights together to ensure that the final converged weights are already quantized. To use NVIDIA Blackwell GPUs, we need a different 4-bit data format: NVFP4. NVFP4 is a new microscaling data format with excellent performance and accuracy on Blackwell GPUs. ✕ Floating point data formats use sign, exponent, and mantissa bits, while integers use sign and value bits. There isn’t much prior art on converting weights from INT4 to NVFP4 directly. Instead, we made a round trip through BF16, first de-quantizing the weights to 16 bits, then running our ordinary quantization script to get back down to 4 bits in NVFP4. The first step, dequantizing from INT4 to BF16, uses the compressed-tensors library to apply scale factors and convert weights from 4-bit integers to 16-bit floating-point numbers. This is a long and compute-intensive process which takes a few hours (you can skip it by using third-party BF16 weights like this release from the team at Unsloth ). From BF16, we were able to follow our ordinary process of converting BF16 to NVFP4 using NVIDIA TensorRT Model Optimizer . We use NVFP4 versus other 4-bit formats thanks to its increased precision from a dual scale factor and its deep support in Blackwell Tensor Cores. While NVFP4 is overall a more precise data format than INT4, switching does not enhance quality as the INT4 quantization was performed during training. There is no way to recover the information that was lost to compression during training. Higher TPS with Tensor Parallelism Once we had the weights in a usable data format, we configured our proprietary inference engine to run the model on a full 8xB200 node. Using all eight of these GPUs effectively requires model parallelism. The model is split across the GPUs, which then communicate with each other over NVLink and NVSwitch interconnects during inference. NVLink, while high bandwidth, is slower than VRAM, so parallelism strategies should minimize the amount of GPU-to-GPU communication required. There are three parallelism strategies to consider for a model like Kimi K2 Thinking: Pipeline Parallelism (PP): Splits a model's layers across multiple GPUs, creating a sequential pipeline where each GPU processes its layers before passing the active request to the next GPU.

Tensor Parallelism (TP): Shards individual tensors and their operations within layers across multiple GPUs, enabling parallel execution of large matrix computations for a single layer.

Expert Parallelism (EP): Distributes whole experts for MoE models into each GPU, where each GPU contains multiple experts.

The Baseten Inference Stack supports all of these forms of parallelism (and several more), as well as mixing and matching parallelism strategies within a single engine. ✕ The Baseten Runtime blends tensor parallelism and expert parallelism along with other parallelism techniques to serve large models efficiently Given the size of Kimi K2 Thinking, some inference setups would require multi-node inference , or more than 8 GPUs. These setups use PP and/or EP across nodes to maximize throughput and to send less data across relatively InfiniBand node-to-node...