Fastest Lowest Cost Qwen3 Asr Batch Transcription

The fastest, lowest-cost batch transcription with Qwen3-ASR Live webinar Oct 28: How to evaluate open models for your coding agents . Save your spot

Model performance

The fastest, lowest-cost batch transcription with Qwen3-ASR

Transcribe an hour of audio with Qwen3-ASR for 1/10th of a cent, with 7x higher throughput, 6.9x lower latency, and no meaningful effect on quality.

Authors

Hasan Khan

Ansel Erol

Published October 28, 2026

Share

Published October 28, 2026

TL;DR Qwen3-ASR 1.7B sits on the Pareto frontier of Hugging Face's Open ASR Leaderboard , but serving it efficiently for batch transcription requires careful optimization. We combined native audio decoding, KV cache quantization, and reduced unnecessary host-to-device syncs to serve 7x more audio per second at high concurrency on a single RTX PRO 6000. On hour-long files, we sustain over 3,000x real-time factor (RTF) at about 1/10th of a cent per audio hour, with no meaningful change in word error rate.

Speech-to-text splits into two very different workloads. On the one hand, real-time transcription is about latency: a voice agent needs words back within a few hunopdred milliseconds of someone speaking. On the other, batch transcription is about throughput: podcast networks, meeting-notes products, and call-center archives have hours of recorded audio and want it transcribed as cheaply as possible. That means getting as many hours of audio through each GPU as it can handle. We optimized Qwen3-ASR 1.7B on Baseten to achieve the lowest cost per audio minute across all transcription providers. Compared with out-of-the-box vLLM, the optimized stack transcribes 7x more audio per second at high concurrency. ✕ Throughput vs. clip length for Qwen3-ASR 1.7B on vanilla vLLM versus Baseten's optimized stack, on a single RTX PRO 6000 at 64 concurrent requests. On vanilla vLLM, throughput plateaus around 380x real-time factor (RTF), and audio files over ten minutes don’t fit in one request. The optimized stack keeps climbing to 2,645x on 5-minute clips, and 3,128x on hour-long files. Qwen3-ASR 1.7B: The most accurate open transcription model Qwen3-ASR 1.7B is the Qwen team's open-weight speech recognition model, released under an Apache 2.0 license. It supports more than 30 languages and 22 Chinese dialects, and it sits on the Pareto frontier of the Open ASR Leaderboard with a 5.4% average word error rate (WER). Our real-time deployment of Qwen3-ASR 1.7B also has the highest quality (i.e., the lowest WER) and the lowest latency on Coval's speech-to-text benchmark , which includes leading transcription models across the market, open and closed. ✕ Word error rate (WER) vs. model size for open-weight models on the Open ASR Leaderboard. Qwen3-ASR 1.7B (5.40%) sits on the Pareto frontier, within 0.07 points of NVIDIA Nemotron 3.5 Transcribe 30B-A4B (5.33%), the most accurate model, at a fraction of its size. Qwen3-ASR has been served for batch transcription on Baseten since the day it launched. Here’s how we made it faster than the vanilla implementation. Our optimized Qwen3-ASR stack We started with vLLM's OpenAI-compatible transcriptions endpoint, which chunks audio into 30-second windows and batches them through the model. From there, we profiled every stage of a request under load, honing in on four things specifically: Native audio decoding

KV cache quantization (without quality loss)

Host-to-device syncs

Request handling

Below are some of the key changes that made our 7x+ higher throughput possible. Native audio decoding Before any model work happens, the server must convert the uploaded M4A or MP3 file into 16 kHz mono samples. By default, vLLM's audio loader uses PyAV to decode audio one frame at a time in Python, resampling on the fly. Each frame step holds Python's global interpreter lock, so the decode threads are serialized rather than running in parallel. This creates a CPU bottleneck at high concurrency, while the GPU sits idle. We replaced the loader with torchcodec , which hands the whole file to FFmpeg in a single native call that decodes, resamples, and downmixes before returning a tensor. This asynchronous approach releases the interpreter lock while the codec processes the request, with identical output to PyAV. For long files, we go one step further and decode and downmix files parallelized across time ranges. Each range is processed with a small overlap, ensuring they splice back together with bit-exact correctness. With these changes, we can preprocess an hour of audio in well under a second. ✕ Decode time per one-minute file for vanilla vLLM's Python decoder versus Baseten's native decoder, measured server-side. The Python decoder takes 76 ms on its own, but 2,195 ms per file when 8 files decode at once, because the threads serialize on the interpreter lock. In contrast, the native decoder takes 45 ms alone and 48 ms with 8 at once, so decode throughput scales with the number of threads. KV cache quantization At high concurrency, attention over the KV cache is the largest share of GPU time. Decode attention is memory-bound: every step reads every cached key and value for every running sequence, and with 1,000 30-second windows in flight, that is the whole cache, every step. Storing the KV cache in FP8 halves the bytes each step reads and doubles the number of sequences that fit in GPU memory. One challenge was that vLLM's default FlashAttention-2 backend does not support FP8 cache on RTX PRO 6000, motivating us to use FlashInfer kernels instead. FlashInfer FP8 improved memory utilization for the attention kernel from 50% to 80%. We kept the weights in BF16 because weight quantization increased word error rate (WER) on our test sets. While the FP8 cache is also lossy, the effect on accuracy is minimal since the attention computation remains in full precision, and each layer recomputes its cache from full-precision activations. ✕ Precision chosen per stage in the optimized decoder. The audio encoder and decoder weights stay in BF16, since the encoder is a small share of GPU time and FP8-quantized weights affected accuracy. The KV cache, the largest source of memory traffic, moves to FP8, which requires half the bytes per step and enables twice the sequences in flight. The attention math runs at full precision and is recomputed from full-precision activations every layer, so rounding errors won’t compound. Minimizing host-to-device syncs Under load, GPU...