deepinfra/kv-local-indexer

deepinfra/kv-local-indexer

Language: Rust

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-06-30T00:39:34Z

Pushed: 2026-06-30T01:11:44Z

Default branch: master

Fork: no

Archived: no

README:

kv-local-indexer

A thin PyO3 wrapper around Dynamo's LocalKvIndexer. It gives a TRT-LLM KV-event worker an in-process radix tree + ring buffer of recent KV events, so the standalone global indexer can recover missed events (Events) or pull a full snapshot (TreeDump) directly from the worker. Replaces the old fixed-window ZMQ replay deque.

Consumed by deepinfra/TensorRT-LLM (tensorrt_llm/serve/kv_zmq_publisher.py), which pip installs the prebuilt wheel from this repo's GitHub Releases. That repo contains no Rust — all of it lives here.

What it exposes

LocalIndexer(worker_id, kv_block_size, max_buffer_size=10000) with:

• apply_stored(seq, token_ids, block_hashes, parent_hash=None, dp_rank=0, lora_name=None, is_eagle=None)
• apply_removed(seq, block_hashes, dp_rank=0)
• apply_cleared(seq, dp_rank=0)
• get_events_json(start=None, end=None) -> str (the recovery query)
• shutdown()

seq is used directly as the event's event_id (one event per ZMQ batch).

Dependency on Dynamo

Depends on dynamo-kv-router, pinned in Cargo.toml to a public upstream git commit of https://github.com/ai-dynamo/dynamo.git:

rev = "b2b7868090a22e02a0cf5733eea7d53e8e0f0829" # ai-dynamo/dynamo main

No local dynamo checkout — cargo fetches that commit during the build. Built with default-features = false, which compiles only the in-process indexer + protocols. The standalone-indexer feature is OFF, so the deepinfra-fork changes (which live entirely under that feature) are not compiled here; upstream at this rev is byte-identical to the fork for what this wrapper needs. To bump it, change rev, rebuild (regenerates Cargo.lock), commit both.

The wheel is portable

The compiled extension links only the standard C runtime (libc / libm / libgcc_s + the loader) — no libpython, no libstdc++, nothing engine-specific. So the wheel runs in any Linux with a glibc at least as new as the build box's. build_wheel.sh builds in manylinux_2_28 (glibc floor 2.28); the engine image is glibc 2.39 / cp312, well above it.

Because it needs nothing from the private localhost:30500 registry, the build can run anywhere, including plain GitHub Actions CI.

Build & release

./build_wheel.sh # -> dist/kv_local_indexer-*.whl (portable, cp312)
./release.sh v0.1.0 # build + publish as a GitHub Release on this repo

release.sh needs gh logged in to *publish* (you, the maintainer). The repo is public, so consumers download the wheel with no auth at all.

For local dev in an active venv:

maturin develop --release # build + install into the active venv

How TensorRT-LLM consumes it

The engine never builds Rust. Dockerfile.python installs the wheel straight from this repo's public release URL (a KV_WHEEL_URL build arg) — no auth, no fetch step, nothing committed in the engine repo.

• Engine Python-only change: nothing here changes; the same wheel URL is reused.
• Rust change here: PR + ./release.sh vX → point KV_WHEEL_URL in the

engine's Dockerfile.python at the new release → rebuild the engine image.

What's committed

• Source (src/lib.rs, Cargo.toml, build glue).
• Cargo.lock — pins every transitive dep for reproducible builds.

The wheel is not committed — it ships via GitHub Releases (see dist/ is gitignored).