mistralai/Voxtral-Mini-4B-Realtime-2602

Voxtral Mini 4B Realtime 2602

Voxtral Mini 4B Realtime 2602 is a multilingual, realtime speech-transcription model and among the first open-source solutions to achieve accuracy comparable to offline systems with a delay of **= 3600 / 0.8 = 45000. In theory, you should be able to record with no limit; in practice, pre-allocations of RoPE parameters among other things limits --max-model-len`. For the best user experience, we recommend to simply instantiate vLLM with the default parameters which will automatically set a maximum model length of 131072 (~ca. 3h).

• We strongly recommend using websockets to set up audio streaming sessions. For more info on how to do so, check [Usage](#usage).
• We recommend using a delay of 480ms as we found it to be the sweet spot of performance and low latency. If, however, you want to adapt the delay, you can change the "transcription_delay_ms": 480 parameter

in the tekken.json file to any multiple of 80ms between 80 and 1200, as well as 2400 as a standalone value.

Benchmark Results

We compare Voxtral Mini 4B Realtime to similar models - both offline models and realtime. Voxtral Mini 4B Realtime is competitive to leading offline models and shows significant gains over existing open-source realtime solutions.

Fleurs

| Model | Delay | AVG | Arabic | German | English | Spanish | French | Hindi | Italian | Dutch | Portuguese | Chinese | Japanese | Korean | Russian | |-----------------------------------------|-------------|---------|--------|--------|---------|---------|--------|--------|---------|-------|------------|---------|----------|--------|---------| | Voxtral Mini Transcribe 2.0 | Offline | 5.90% | 13.54% | 3.54% | 3.32% | 2.63% | 4.32% | 10.33% | 2.17% | 4.78% | 3.56% | 7.30% | 4.14% | 12.29% | 4.75% | | Voxtral Mini 4B Realtime 2602 | 480 ms | 8.72% | 22.53% | 6.19% | 4.90% | 3.31% | 6.42% | 12.88% | 3.27% | 7.07% | 5.03% | 10.45% | 9.59% | 15.74% | 6.02% | | | | | | | | | | | | | | | | | | | | 160 ms | 12.60% | 24.33% | 9.50% | 6.46% | 5.34% | 9.75% | 15.28% | 5.59% | 11.39%| 10.01% | 17.67% | 19.17% | 19.81% | 9.53% | | | 240 ms | 10.80% | 23.95% | 8.15% | 5.91% | 4.59% | 8.00% | 14.26% | 4.41% | 9.23% | 7.51% | 13.84% | 15.17% | 17.56% | 7.87% | | | 960 ms | 7.70% | 20.32% | 4.87% | 4.34% | 2.98% | 5.68% | 11.82% | 2.46% | 6.76% | 4.57% | 8.99% | 6.80% | 14.90% | 5.56% | | | 2400 ms | 6.73% | 14.71% | 4.15% | 4.05% | 2.71% | 5.23% | 10.73% | 2.37% | 5.91% | 3.93% | 8.48% | 5.50% | 14.30% | 5.41% |

Long-form English

| Model | Delay | Meanwhile ( [!Tip] > We've worked hand-in-hand with the vLLM team to have production-grade support for Voxtral Mini 4B Realtime 2602 with vLLM. > Special thanks goes out to Joshua Deng, Yu Luo, Chen Zhang, Nick Hill, Nicolò Lucchesi, Roger Wang, and Cyrus Leung > for the amazing work and help on building a production-ready audio streaming and realtime system in vLLM.

> [!Warning] > Due to its novel architecture, Voxtral Realtime is currently only support in vLLM. We very much welcome community contributions > to add the architecture to Transformers and Llama.cpp.

We've worked hand-in-hand with the vLLM team to have production-grade support for Voxtral Mini 4B Realtime 2602 with vLLM. vLLM's new Realtime API is perfectly suited to run audio streaming sessions with the model.

Installation

Make sure to install vllm from the nightly pypi package. See here for a full installation guide.

uv pip install -U vllm

Doing so should automatically install `mistral_common >= 1.9.0`.

To check:

python -c "import mistral_common; print(mistral_common.__version__)"

You can also make use of a ready-to-go docker image or on the docker hub.

Make sure to also install all required audio processing libraries:

uv pip install soxr librosa soundfile

Also we recommend using Transformers v5 as v4 can clutter the terminal with unnecessary warnings (see here)

uv pip install --upgrade transformers

Serve

Due to size and the BF16 format of the weights - Voxtral-Mini-4B-Realtime-2602 can run on a single GPU with >= 16GB memory.

The model can be launched in both "eager" mode:

VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 --compilation_config '{"cudagraph_mode": "PIECEWISE"}'

Additional flags:

• You can set --max-num-batched-tokens to balance throughput and latency, higher means higher throughput but higher latency.
• You can reduce the default --max-model-len to allocate less memory for the pre-computed RoPE frequencies,

if you are certain that you won't have to transcribe for more than X hours. By default the model uses a --max-model-len of 131072 (> 3h).

Client

After serving vllm, you should see that the model is compatible with vllm's new realtime endpoint:

...
(APIServer pid=3246965) INFO 02-03 17:04:43 [launcher.py:58] Route: /v1/realtime, Endpoint: realtime_endpoint
...

We have added two simple example files that allow you to:

• Stream audio files
• Simple gradio live transcription demo

…