Delivering Fast Real Time Text To Speech With Gradium

Delivering real-time text-to-speech with Gradium Use your OpenAI commitments on Baseten open models . Learn more

AI models

Delivering real-time text-to-speech with Gradium

Achieve natural-sounding conversational experiences using Gradium TTS on Baseten.

Authors

Albert Lee

Published October 13, 2026

Share

Published October 13, 2026

TL;DR Gradium TTS is built for real-time voice agents, achieving TTFA faster than typical human response times through its Delayed Streams Modeling architecture and GPU optimizations, including kernel fusion and CUDA Graphs. It is available in Baseten's Model Library now as a dedicated deployment.

Gradium TTS: TTFA faster than human speech Gradium’s text-to-speech (TTS) model converts written text into spoken audio and is designed specifically for real-time AI voice agent interactions. Time-to-First-Audio (TTFA) measures the duration between sending a TTS request and receiving the first playable audio. It is critical for natural-sounding conversations. To deliver reliably natural conversations, Gradium TTS achieves a TTFA faster than human speech response times.

Gradium latency race demo

Try the TTS Latency Race demo yourself and access its code repository here . Independent Coval benchmarks place Gradium TTS in the faster-than-human-speech tier, giving voice agents time to reason or retrieve context without breaking conversational flow. On the Coval TTS benchmark (as of September 2026), Gradium TTS achieved a median TTFA of 214 ms with an interquartile range tight enough that developers can set reliable latency budgets across the rest of their pipeline. Gradium's TTS doesn't wait for the full sentence Gradium TTS is streaming: it can start speaking before the upstream text generator, such as an LLM, has finished producing the full response. Specifically, Gradium's models use Delayed Streams Modeling (DSM) , a framework developed by Kyutai that treats text and audio as two parallel streams running on a shared clock, so audio can be generated before text has finished arriving. Both streams align to the same frame rate, one step every 80 ms, so the model can generate audio as text arrives. The key idea of DSM is the audio stream runs a few steps behind the text, giving the model a glimpse of what comes next, similar to how a human speaker adjusts tone, inflection, and pacing based on the words they're about to say. A longer delay gives the model more context and better voice quality, while a shorter one lowers latency. Audio output follows text by a tuned number of decoding steps ✕ Delayed Streams Modeling aligns text and audio to a shared 80 ms clock and runs the audio stream a fixed number of steps behind the text, so each audio frame is generated with the next few words already in view. Gradium TTS starts speaking after the first few tokens, not the full token generation. The model's design also keeps each 80 ms frame quick to generate, maintaining low latency once speech begins. How each 80 ms audio frame gets built Each step in the audio stream is produced by two transformers with distinct roles: Temporal transformer (the "reader"): Moves through the streams one step at a time, taking in the newest text token and the previous audio frame. It produces one vector describing what the next frame of audio should sound like, and signals when it's ready for the next word, which keeps the speech naturally paced when text arrives unevenly.

Depth transformer (the "refiner"): Turns that vector into an audio frame. Each frame is encoded as up to 32 tokens, arranged so the first captures the most important features and each later one adds finer detail. The refiner generates them in that order, so the frame becomes progressively more accurate with every token.

One large read, one small refinement loop ✕ Within each 80 ms step, the temporal transformer reads the newest text token and the previous audio frame and produces a single summary vector. The depth transformer then generates the frame's audio tokens one at a time, coarse to fine. This split enables Gradium TTS's low TTFA. The larger temporal transformer, which has to track context, runs only once per step, while the smaller depth transformer runs up to 32 times per step but stays cheap because it never looks back in time. Those many small refiner passes also benefit from additional GPU optimization. GPU optimization specifically for smaller, latency-optimized models GPUs do the heavy computation for AI models, but they don't work alone. The CPU acts as the dispatcher, and for each computational step, it selects the right GPU kernel, prepares its inputs, and issues the command to launch it. Every one of those handoffs takes a small, fixed amount of time. In large language models, the GPU is the bottleneck, and each GPU kernel runs long enough that the CPU can queue the next launch before the current one finishes. However, in smaller AI models like streaming TTS, GPU kernels complete faster than the CPU can queue them. As a result, the CPU becomes the latency bottleneck, leading to lower GPU utilization. Gradium addresses this challenge in two ways: Kernel Fusion : Using torch.compile and Triton, Gradium merges operations that previously required several separate GPU kernel launches into a single kernel, so each launch does more work.

CUDA Graphs : The full sequence of kernels for a forward step is recorded once. On every subsequent step, the CPU issues a single replay command and the GPU driver runs the entire recorded sequence back-to-back, rather than waiting for each kernel to be launched individually.

Typical kernel launches versus CUDA Graph Replay ✕ With CUDA Graphs, the CPU issues one replay command and the recorded kernel sequence runs back-to-back, reducing TTFA. In Gradium's published benchmarks , enabling CUDA Graphs cut median TTFA by more than 2.5x at small batch sizes. Scaling Gradium TTS on Baseten Traffic to voice AI applications tends to be bursty. There is high demand at the start of business hours and during triggering events such as outages or marketing campaigns. Required concurrency can rise and fall in minutes, so provisioning for the peak leaves GPUs idle most of the day, while provisioning for the average degrades performance during the first minutes of every spike. Autoscaling resolves this, but a new replica can't serve calls until its model weights are loaded into GPU memory. Weights are typically pulled...