inclusionAI/Realtime-Venus

inclusionAI/Realtime-Venus

Language: Python

License: Apache-2.0

Stars: 14

Forks: 1

Open issues: 1

Created: 2026-09-16T07:57:55Z

Pushed: 2026-09-19T06:47:56Z

Default branch: main

Fork: no

Archived: no

README:

Overview

See, listen, and respond in real time—with background tasks running alongside the conversation. Realtime-Venus combines full-duplex conversational models with an asynchronous execution framework. You can ask the assistant to work on a task and continue talking while it runs. The result returns to the same conversation for spoken delivery.

The project brings together three components:

| Component | Role | | --- | --- | | Realtime-Venus-Omni · 9B | A conversational model for streaming audio and video, proactive interaction, and native speech generation. | | Realtime-Venus-Audio · 9B | A separately trained model for spoken interaction, audio understanding, and native speech generation. | | Realtime-Venus-Harness | A shared runtime that captures delegated requests, executes background work, and returns results to the originating conversation. |

What Realtime-Venus enables

• Continuous audiovisual interaction. Omni observes streaming camera and microphone input and can initiate a response when the evolving scene calls for it.
• Full-duplex conversation. The models process incoming speech while speaking, learning to distinguish acknowledgments and background speech from interruptions that call for a revised response.
• Asynchronous delegation. A model can hand a natural-language task to Harness for reasoning or tool execution while live interaction continues.
• Results in the same conversation. Harness prepares a reply; the conversational model handles its timing and native speech output. Work tracking separates task completion from spoken delivery.

The paper describes the model family, dual-loop runtime, training data, and evaluation. This source release includes the Omni model integration, the reusable Harness package, and a browser demo using a Codex task backend. The demo's microphone-only mode also uses Omni.

Quick start

1. Install dependencies

Standalone inference requires Python 3.10, CUDA, and FFmpeg. With the Hugging Face CLI or ModelScope CLI installed, run one of the following commands from the source repository root to download the Omni and Audio checkpoint directories:

huggingface-cli download inclusionAI/Realtime-Venus --local-dir . \
--include "Realtime-Venus-Omni/*" "Realtime-Venus-Audio/*"
# or:
modelscope download --model inclusionAI/Realtime-Venus --local_dir . \
--include "Realtime-Venus-Omni/*" "Realtime-Venus-Audio/*"

Then install the dependencies for your model:

# Realtime-Venus-Omni
python -m pip install -r Realtime-Venus-Omni/requirements.txt

# Realtime-Venus-Audio
python -m pip install -r requirements.txt

For the browser demo, run the installer on Linux. It creates an isolated Python 3.11/3.12 environment and installs the model, web service, and Harness dependencies:

bash install.sh

Then choose standalone inference or the online experience below.

2. Frontend Usage

Download the model checkpoint from the links above and replace the example paths with your local directories.

Realtime-Venus-Omni

Offline video chat. Generate text and speech responses based on the input video. The two examples demonstrate basic video chat and chat with long-video memory.

export REALTIME_VENUS_MODEL_PATH=/path/to/Realtime-Venus-Omni

python frontend/Realtime-Venus-Omni/offline_chat.py
# With long-video Memory
python frontend/Realtime-Venus-Omni/offline_memory_chat.py

Full-duplex video interaction. Stream the visual and audio content of a recorded video into the model as it generates text and speech responses. The three examples demonstrate text questions, spoken questions, and long-video memory.

python frontend/Realtime-Venus-Omni/duplex_chat.py
python frontend/Realtime-Venus-Omni/duplex_speech_in_chat.py
python frontend/Realtime-Venus-Omni/duplex_memory_chat.py

Duplex examples save subtitled videos. See the [Omni guide](frontend/Realtime-Venus-Omni/README.md) for inputs and output paths.

Realtime-Venus-Audio

Offline audio understanding. Provide a complete audio clip for the model to understand and respond to in text.

python frontend/Realtime-Venus-Audio/audio_offline_chat.py \
--model-path /path/to/Realtime-Venus-Audio \
--audio frontend/Realtime-Venus-Audio/case/case_offline.wav

Full-duplex audio interaction. Stream recorded audio into the model and generate speech responses while it continues listening. Save the output as a 24 kHz WAV file.

python frontend/Realtime-Venus-Audio/audio_duplex_chat.py \
--model-path /path/to/Realtime-Venus-Audio \
--audio frontend/Realtime-Venus-Audio/case/case_duplex.wav \
--output output/duplex_response.wav

Both scripts accept --system-prompt and --prompt; see the [Audio guide](frontend/Realtime-Venus-Audio/README.md) for decoding options.

3. Talk to Realtime-Venus

The online Duplex demo requires at least one NVIDIA A100 GPU. Place the complete Omni checkpoint in model_weight/, then start the demo:

bash start.sh

For an existing checkpoint elsewhere:

bash start.sh --model-path /path/to/Realtime-Venus-Omni

Complete Codex login if prompted. The launcher starts the model API on 8031 and the browser service on 8032.

If running on a remote server, keep this tunnel open on your local computer, replacing user@server with your SSH login:

ssh -N -L 8032:127.0.0.1:8032 user@server

Open http://localhost:8032, select voice, camera + microphone, or video upload, and start a conversation. Checkpoint layout, configuration, and service management are covered in the [Demo guide](demos/README.md).

Results

Figure 1. Video and audio understanding results from the paper.

Figure 2. Full-duplex interaction results from the paper.

Code map

Realtime-Venus/
├── harness/ # Context, routing, agents, work state, and delivery
│ ├── README.md
│ └── requirements.txt # Harness media dependencies
├── frontend/
│ ├── Realtime-Venus-Omni/ # Audiovisual inference examples
│ └── Realtime-Venus-Audio/ # Audio inference examples and input samples
├── demos/
│ ├── model/ # Checkpoint adapter and model HTTP...