I Spent 31 Hours On The Math Behind Turboquant So You Dont Have To

I spent 31 hours on the math behind TurboQuant so you don't have to Announcing our Series F . Learn more

Model performance

I spent 31 hours on the math behind TurboQuant so you don't have to

A deep dive by Baseten's research team breaking down the math behind TurboQuant

Authors

Ali Taha

Last updated March 27, 2026

Share

How does TurboQuant actually work? Is it worth the hype? Is it any different from modern quantization techniques like Nvidia's FP4? ✕

Conditioning of PolarQuantTo understand TurboQuant, one must first understand PolarQuant: a novel quantization method employing random preconditioning and polar transformation. Our method transforms the KV embeddings into polar coordinates using a recursive algorithm and then quantizes resulting angles. The long-context evaluation demonstrates that PolarQuant compresses the KV cache by over 4.2x That's a lot to take in. Let's break it down. From the very beginning: KV Cache: The problem ✕

Every transformer-based LLM computes attention the same way. For each token, the model produces three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what information do I carry). Attention scores are computed as softmax(Q·K^T/√d)·V : the query dot-products every key to figure out which tokens matter, then takes a weighted sum of the values. ✕

The trick that makes autoregressive generation fast is KV caching. Once a token's key and value vectors are computed, they never change for a given sequence. So you store them and reuse them for every future token in that request. The problem: this cache grows linearly with sequence length. Every new token adds one key vector and one value vector per layer per head. For Llama-3.1-8B with 32 layers, 8 KV heads, head dimension 128, and 128K context: that's 128,000 × 32 × 8 × 128 × 2 (K and V) × 2 bytes (FP16) = 16 GB of KV cache alone. For a single user session. Add a bunch more concurrent sessions on the same GPU and the KV cache becomes the memory bottleneck. ✕

Quantization: Existing solutions Currently, the most aggressive quantization solution is NVFP4 two layer quantization. This was covered in previous posts, but, to give a brief overview, here's how it works: ✕

You scan through the entire matrix. You find the global maximum value in the matrix. You then find the maximum value for each block of 16 elements. This gives you a local and global granularity. You divide the number down by the local maximum, and you divide the scales by the global maximum. You then cast the numbers into the closest 4-bit bucket. You do this for both matrices (weights and activations). You multiply the two, making use of specialized cores, which internally will take your scaling factors, reconstruct them, and multiply the result of your matrix multiplication with the inverse of the scaling factors to yank them back into full-precision. The bucketing problem Every quantization method is, at its core, a mapping. You take a continuous number and assign it to the nearest bucket. With 4 bits you have 16 buckets. With 2 bits you have 4. The problem isn't the mapping itself: it's that you need to know where to place the buckets, and that requires measuring your data first. For each block of values, you compute a scale factor (max value) and a zero-point (offset), store them in full 16-bit precision alongside your quantized values, and use them to reconstruct later. These normalization constants are pure overhead. One outlier corrupts the entire block's precision. ✕

How can we fix bucket problem? What if we knew the distribution of our data ahead of time? What if we could guarantee that all values cluster in a predictable, tight range? But...there exists no way through which we can somehow magically manipulate our data into a tightly clustered distribution to ensure that we can map it effectively. so we pay the price of 2-layered quantization. we accept this mediocrity at face value. PolarQuant The authors at Google Research argue that they *could*, in fact, manipulate the data into a tightly clustered distribution. But how?They state 2 properties of multivariate normal random variables: Multiply any fixed vector by a random matrix with bell curve entries. The output is a multivariate Gaussian centered at zero, with variance equal to the squared length of the original vector.

S ⋅ x ∼ N ( 0 , ∥ x ∥ 2 ⋅ I m ) S⋅x∼N(0,∥x∥2⋅Im) S ⋅ x ∼ N ( 0 , ∥ x ∥2 ⋅ I m ) If every coordinate of a vector is drawn from a standard bell curve, the length of that vector follows a generalized gamma distribution. In high dimensions, this length concentrates tightly around √d.

f R ( r ) = 22 d / 2 ⋅ Γ ( d / 2 ) r d − 1 e x p ⁡ ( − r 2 / 2 ) fR(r)=22d/2⋅Γ(d/2)rd−1exp⁡(−r2/2) f R ( r ) = 22 d /2 ⋅ Γ ( d /2 ) r d − 1 e x p ⁡ ( − r 2/2 ) We will do something more elegant than a proof. We will code. def fact1_gaussian_norms (): torch.manual_seed( 0 ) dims = [ 16 , 64 , 128 , 512 ] n_samples = 10000 fig, axes = plt.subplots( 1 , len (dims), figsize=( 5 * len (dims), 5 )) for ax, d in zip (axes, dims): x = torch.randn(n_samples, d) norms = torch.norm(x, dim= 1 ).numpy() ✕

Distribution proof of fact 1 1 tokenizer = AutoTokenizer.from_pretrained("gpt2") 2 model = GPT2LMHeadModel.from_pretrained("gpt2") 3 model.eval() 4 text = "Baseten has the best performance engineers" 5 input_ids = tokenizer.encode(text, return_tensors="pt") 6 7 with torch.no_grad(): 8 outputs = model(input_ids, use_cache=True) 9 K = outputs.past_key_values[5][0][0] 10 11 x = K[0, 5] 12 d = x.shape[0] 13 norm_x = torch.norm(x).item() 14 15 n_trials = 10_000 16 torch.manual_seed(0) 17 18 S_all = torch.randn(n_trials, d, d) # 10K different random matrices 19 y_all = torch.bmm(S_all, x.unsqueeze(0).unsqueeze(-1).expand(n_trials, d, 1)) 20 y_all = y_all.squeeze(-1) Proving the second fact, that for any vector x, if S is a random matrix with i.i.d. normal entries, the vector matrix product has a multivariate normal distribution, is just as simple. We pick one token, in this case head 0, layer 5, token 5. It has 64 dimensions. In this case, this token happens to have a norm of 9.85. We then draw 10k random matrices. We multiply each of these randomized matrices with the vector x. We then analyze the distributions of each dimension. ✕

This proves to us that, after random preconditioning, vectors behave like Gaussians. Every feature of the vector, sampled across all the products, follow the...