meituan-longcat/omni-flow

meituan-longcat/omni-flow

Description: A unified workflow orchestration and distributed KV cache sharing framework for multimodal inference.

Language: Python

Stars: 7

Forks: 0

Open issues: 0

Created: 2026-06-25T08:05:18Z

Pushed: 2026-07-02T02:29:01Z

Default branch: v0.1.0

Fork: no

Archived: no

README: 

A unified workflow orchestration and distributed KV cache sharing framework for multimodal inference.

About

As multimodal models rapidly evolve from text-only to image, audio, and video, inference systems face three core challenges: flexible workflow orchestration across heterogeneous components, low-latency data transfer between roles, and safe KV cache & weight sharing among multiple consumers. Omni-Flow addresses these through a three-layer abstraction:

• Control Flow — Exposes a Python DSL for composing heterogeneous computing units into a unified dataflow graph. It supports both static DAGs and dynamic conditional routing, and ships with built-in service discovery and a suite of load-balancing strategies, making it straightforward to wire up arbitrarily complex multimodal pipelines.
• Data Flow — Provides a distributed KV cache abstraction that transcends the conventional prefill/decode boundary. It unifies cache allocation across roles, delivers direct cross-role transmission over a three-tier paged storage hierarchy (GPU → CPU → SSD), and employs zero-copy channels to minimise end-to-end transfer latency.
• Compute Flow — Implements sophisticated multimodal prefix matching for KV reuse across multi-turn dialogues, and assumes full control of KV cache management and sampling logic through a unified SGLang interface. This lets diffusion models share the same LLM forward path and operate under identical parallel semantics without any bespoke adaptation.

> 📄 See the technical report: arxiv

Quick Start

1. Pull the Base Docker Image

Omni-Flow builds on top of the official SGLang image. Pull the recommended version before proceeding:

docker pull lmsysorg/sglang:v0.5.10.post1-cu130

> Note: Use v0.5.10.post1 (without -cu130) if your NVIDIA driver is older than 570 (CUDA export REDIS_PORT=

export REDIS_PASSWORD= # if auth is enabled

### 4. Launch a Service (DeepSeek-V2 example)

bash omni_flow/examples/deepseek_v2_llm_only/start.sh --foreground

Other launch options:

bash omni_flow/examples/deepseek_v2_llm_only/start.sh # daemon mode bash omni_flow/examples/deepseek_v2_llm_only/start.sh --foreground # foreground debug bash omni_flow/examples/deepseek_v2_llm_only/start.sh --stop # stop bash omni_flow/examples/deepseek_v2_llm_only/start.sh --status # check status

### 5. Send a Request

bash omni_flow/examples/deepseek_v2_llm_only/request.sh

## Supported Models

| Model | Scenario | Example | Description |
|-------|----------|---------|-------------|
| **DeepSeek-V2** | LLM-only streaming inference | [`deepseek_v2_llm_only/`](omni_flow/examples/deepseek_v2_llm_only/README.md) · [README](omni_flow/examples/deepseek_v2_llm_only/README.md) | Single graph, single role, streaming token output |
| **LongCat-Next** | Full-modality conversation | [`longcat_next/`](omni_flow/examples/longcat_next/) · [README](omni_flow/examples/longcat_next/README.md) | LLM + Vision Encoder + Audio Encoder + Image Decoder + Audio Decoder (5 roles) |
| **HunyuanImage-3** | Image generation pipeline | [`hunyuan_image3_pipeline/e2e/`](omni_flow/examples/hunyuan_image3_pipeline/e2e/) · [README](omni_flow/examples/hunyuan_image3_pipeline/e2e/README.md) | AR + Diffusion, LLM Backbone DiT, t2i/ti2i + CFG |

## Deployment

Omni-Flow uses Ray for distributed deployment, with resource topology declared via `resource.yaml`:

- **Elastic scaling**: modify `replicas` to scale within a resource group
- **Layer co-location**: `layer: 0` determines PG shape; `layer ≥ 1` layers onto the same physical GPUs
- **KV Barrier synchronization**: ensures accurate HBM evaluation timing in co-located deployments to prevent OOM

See [`SCHEMA_V2.md`](omni_flow/deploy/launch/SCHEMA_V2.md) and [`launch/README.md`](omni_flow/deploy/launch/README.md) for detailed configuration.

## Contact

For collaboration or hiring inquiries, please contact: **xiaobin14@meituan.com**.

## Citation

@misc{omni-flow2026, title = {Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference}, author = {Bin Xiao and Jingfu Dong and Changran Wang and Yitian Chen and Xiaoyu Zhao and Yuqi Peng and Jianping Lin and Yuchen Xie}, year = {2026}, eprint = {2606.31093}, archivePrefix = {arXiv}, primaryClass = {cs.DC} }