deepinfra/mooncake

deepinfra/mooncake

Description: Mooncake with NIC-free transfer capability

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-10-23T05:00:20Z

Pushed: 2026-10-23T04:29:13Z

Default branch: main

Fork: yes

Parent repository: kvcache-ai/Mooncake

Archived: no

README:

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests while adhering to SLOs.

🔄 Updates

• Oct 10, 2026: TensorRT-LLM merged a native Mooncake Store connector into KVCMv2, enabling content-addressed KV cache pooling and prefix reuse across serving instances.
• Oct 6, 2026: TokenSpeed v0.1.1 includes Mooncake Store as an L3 KV cache backend beneath its host cache, alongside Mooncake Transfer Engine support for PD and EPD disaggregation.
• Sept 17, 2026: vLLM added cross-encoder embedding reuse through Mooncake Store, complementing its encoder-cache transfer connector for multimodal disaggregated serving. Both capabilities are included in vLLM v0.31.0.
• Aug 28, 2026: LMDeploy merged Mooncake Store support, enabling shared KV cache pooling and prefix reuse across PyTorch inference instances. Included in LMDeploy v0.18.0.
• Aug 24, 2026: AgentX / InferenceX v3 highlights Mooncake's contributions to agentic inference workloads, including ROCm wheels, CI, and packaging improvements. Mooncake optimization details.

More

• Aug 20, 2026: Mooncake is integrated into Miles as a rollout data-transfer backend for the fragmented, heterogeneous data moving between rollout and training in disaggregated RL. Blogs: KVCache.AI, lmsys.
• Aug 17, 2026: Mooncake is integrated into Speculators as a distributed backend for multi-node online training, efficiently moves hidden-state between vLLM inference workers and Speculators trainers through RDMA, eliminating the need for massive hidden-state storage in offline training. Benchmark on GB300 NVL72.
• Aug 5, 2026: llm-d added a Mooncake PD deployment guide, complementing its Mooncake Store tiered offloading integration for vLLM serving on Kubernetes.
• Aug 1, 2026: Dynamo merged an event-driven Mooncake shared-cache index for SGLang HiCache routing, using Mooncake object events to track reusable KV pages in the shared pool.
• Jul 24, 2026: FlexKV added Mooncake Store as a distributed REMOTE-tier backend, enabling content-addressed KV cache sharing alongside its existing Mooncake Transfer Engine integration.
• Jul 12, 2026: TransferQueue added GPUDirect RDMA support for its Mooncake Store backend, enabling GPU clients to exchange RL tensors with the host-memory store through GDR.
• May 7, 2026: 🚀 vLLM officially features Mooncake Store — a deep dive into how Mooncake's distributed KVCache engine supercharges vLLM inference with high-throughput, memory-efficient, cross-instance KV cache sharing!
• Apr 29, 2026: SGLang introduces RDMA-based P2P weight transfer for large-scale distributed RL using Mooncake TransferEngine, achieving 7x faster weight updates for the 1T-parameter Kimi-K2 model (53s → 7.2s) with zero-copy RDMA transfer across thousands of GPUs.
• Apr 3, 2026: LMCache introduced a native Mooncake Store L2 adapter for multiprocess mode, connecting the LMCache MP server to the distributed KV cache pool.
• Apr 1, 2026: The TENT technical report introduces declarative data-movement orchestration for heterogeneous interconnects, with evaluations in SGLang HiCache and checkpoint-engine workloads.
• Mar 19, 2026: TorchSpec: Speculative Decoding Training at Scale is open sourced, using Mooncake to decouple inference and training via efficient hidden states management.
• Mar 16, 2026: FastDeploy added Mooncake Store cache read/write support to its PD pipeline, complementing its global KV cache pooling across inference instances.
• Mar 5, 2026: LightX2V now supports disaggregated deployment based on Mooncake, enabling encoder/transformer service decoupling with Mooncake Transfer Engine for high-performance cross-device and cross-machine data transfer. Details in blog.
• Mar 3, 2026: verl added a Mooncake checkpoint-engine backend for peer-to-peer weight synchronization between training and rollout workers.
• Feb 25, 2026: SGLang merged Encoder Global Cache Manager, introducing a Mooncake-powered global multimodal embedding cache that enables cross-instance sharing of ViT embeddings to avoid redundant GPU computation.
• Feb 24, 2026: vLLM-Omni introduces disaggregated inference connectors with support for both...