Under the Hood: Serving Kimi K3

Under the Hood: Serving Kimi K3 | DigitalOcean

© 2026 DigitalOcean, LLC. Sitemap .

Engineering

Under the Hood: Serving Kimi K3

By Jonathan Dieu and Shree Murthy

Updated: July 30, 2026 12 min read

<- Back to blog home

DigitalOcean launched Kimi K3 on day 0. It’s already one of the most popular models on the platform and across the market: second most likes on Hugging Face, sixth most traffic on OpenCode. Getting a model this size running well on day zero took real work across several teams. Thanks to Moonshot AI, Inferact, RadixArk, NVIDIA, and AMD for the help getting there.

Standing up a new model, integrating it into DigitalOcean’s Inference Engine, and showcasing its unique attributes on day 0 takes three things: the right hardware, a tuned serving stack, and rigorous verification against Moonshot’s own benchmarks.

Here are the lessons we learned along the way:

Hardware selection and implementation

We selected NVIDIA HGX™ B300 and AMD Instinct™ MI350x GPUs to run K3 because these instances provide the memory capacity, FLOPs, and interconnect horsepower necessary for a model of K3’s size and architecture. We built our distributed inference stack with llm-d because it includes native support for GPU type heterogeneity. This let us quickly onboard K3 to both AMD and NVIDIA platforms.

Kimi K3 has roughly 2.78 trillion total parameters , 896 routed experts, and an attention stack that interleaves 69 Kimi Delta Attention (KDA) layers with 24 Gated Multi-head Latent Attention (MLA) layers. Kimi-K3 weights are ~1.56 TB in total, which requires about 195 GiB per GPU.

Given such a large memory footprint for the weights alone, and a need to keep enough headroom for KV cache and activations, the practical unit of deployment is an 8x NVIDIA HGX B300 or AMD Instinct MI350X server. Both have 288GB of VRAM capacity, and after loading the weights, there is still some amount of practical memory left for the KV cache.

Entire weights cannot be loaded on a single GPU. That’s where the high-speed scaled-up NVIDIA’s NVLink or AMD’s Infinity Fabric is critical to ensure there is enough interconnect horsepower for bandwidth intensive, latency sensitive attention and expert parallel computations.

Model Optimization

The K3 serving recipe was optimized in collaboration with the vLLM team along the following dimensions:

Throughput scaling

We maximized input/output token throughput without sacrificing unit economics. Concretely, that meant three things:

A tuned memory budget. We fine-tuned gpu-memory-utilization , and MXFP4 quantization keeps Kimi K3’s ~1.4 TiB of weights small enough to leave memory headroom for KV cache.

A higher batching ceiling. We raised max-num-batched-tokens so each scheduling step can pack more tokens per batch, lifting throughput per GPU without adding nodes.

A faster prefill path. Prefill runs through TensorRT-LLM’s ragged MLA attention backend with prefill-query quantization—a meaningfully faster prefill path than the default for a model this size.

Latency and concurrency

We looked for a concurrency sweet spot: enough simultaneous requests to keep token volume healthy, without letting latency degrade the user experience. Concretely, that meant three things:

Staggered TTFT targets by prompt length . Rather than one flat TTFT SLA for every request, we set targets that scale with input length—from short prompts up to 1M-token inputs. We found that this matches real-world, mixed-workload traffic far better than a single flat target.

Workload-specific ITL SLAs . Chat and agentic workloads have different inter-token latency needs, so we set separate ITL targets for each.

TPS and E2E latency held to reasonable bounds throughout, so gains in throughput wouldn’t come at the user’s expense.

Infrastructure readiness

Serving a large model like Kimi-K3 on 8x servers means that the number of concurrent user sessions per server will be relatively low. Once we identified the concurrency sweet spot, along with supporting 1 million token context windows, we had enough data points to plan forNVIDIA HGX™ B300 and AMD Instinct™ MI350X capacity needed for the launch.

Model Verification

An open-weight model’s benchmark scores are only real if whoever serves it configures the serving stack correctly. A closed frontier model like Fable 5 or GPT 5.6 is tuned, served, and validated end-to-end by one vendor. Kimi K3 doesn’t get that: Moonshot publishes the weights, and every infra provider serving them, DigitalOcean included, has to independently get every decoding parameter, every parsing rule, and every wire-format detail right. If we get any of it wrong, the model underperforms its own published numbers—not because the weights are worse, but because the serving layer is.

Kimi K3 is a near-frontier model, competitive with Fable 5 and GPT 5.6 on real benchmarks . But that’s only true if the model is served correctly. Bad serving can turn a frontier model into a mediocre one. After previous model releases, Moonshot found that vendors serving identical weights were getting inconsistent benchmark results, sometimes due to decoding parameter mismatches, sometimes due to deeper serving-stack issues. That’s why they open-sourced the Kimi Vendor Verifier (KVV) alongside K2.6, to make sure every vendor serving their weights runs them correctly.

Moonshot open-sourced the Kimi Vendor Verifier (KVV) project alongside K2.6 for exactly this reason: they’d already learned that open-sourcing weights is only half the job. The other half? Making sure every vendo r serving those weights runs them correctly.

Their conclusion, in their own words : the more open the weights and the more diverse the deployment channels, the less controllable quality becomes. If users can’t distinguish a model capability defect from an engineering implementation deviation, trust in the entire open-weight ecosystem erodes. Thus, KVV is Moonshot’s answer: a six-benchmark suite that every vendor has to pass, which includes decoding-param pre-verification, OCRBench, MMMU Pro, AIME2025, a tool-calling F1 and JSON-schema-accuracy benchmark, and SWE-Bench. Ongoing fixes contributed directly to vLLM, SGLang, and KTransformers, and pre-release access so vendors can catch problems before users do, with a public leaderboard of vendor results.

Every test in the KVV—tool-call schema injection, dynamic tools, streaming spec compliance, decoding-param handling—exists because a misconfigured serving stack can turn a...