Fireworks Quantization

How Fireworks evaluates quantization precisely and interpretably

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Fireworks Quantization How Fireworks evaluates quantization precisely and interpretably

PUBLISHED 8/1/2024

Table of Contents

TL;DR

Intro

KL Divergence - Evaluating Quantization Quality

Ensuring Quality

Issues with other evaluation methods for quantization quality

Conclusion

Table of Contents

TL;DR

• There’s no one size fits all for quantization - There are a variety of quantization techniques and potential parts of a model to quantize. Fireworks works closely with individual customers, like Cursor and Superhuman, to tailor quantization for individual use cases • KL divergence for measuring quantization quality - To measure quantization quality, we prefer divergence metrics - they’re highly interpretable, accurate and supported by literature • Other evaluation methods - Fireworks carefully evaluates our models using both divergence and task-based metrics to ensure quality matches reference models. However, we advise against using task-based approaches to measure quantization quality because high noise limits precision • You’re the best judge of quality - Different quantization techniques affect use cases differently. Therefore, end developers are ultimately the best judge of quantization quality. We’re excited about Fireworks’ industry leading combination of speed, cost and quality and encourage you to judge for yourself!

Intro

With the release of Llama 3.1, there’s been considerable discussion about the benefits and tradeoff of different quantization methods. Evaluating quantization quality is notoriously tricky, so in this post, we share how Fireworks approaches quantization and evaluates tradeoffs. With both general LLM inference and quantization, we believe there’s no one-size-fits all solution for LLM inference. Inference and quantization set-ups are ideally tailored specially for a certain use case. A common misunderstanding with quantization is that it’s black and white and that a model is either quantized (well) or not. However, there are both A wide range of possible quantization techniques , with varying levels of aggression. For example, we may use techniques such as SmoothQuant , GPTQ , and outlier reducing transforms ( Hadamard , SpinQuant ) to maximize the quality of the quantized model. Granularity of scaling factors may vary from one-per-entire-tensor to one-per-small-group of values. At runtime, the quantized model may use techniques such as online hadamard to reduce KV cache incoherence, or online scales for linear transformations . Various layers/parts of the model that may be quantized, like QKV projection, attention or KV cache. Some layers may be skipped or not.

Generally, as you increase quantization aggression level, performance improves and quality degrades. However, it’s possible to achieve disproportionate performance improvements with an insignificant effect on quality. The tradeoff between quality and performance varies based on factors including: • (a) Specific model (or even model fine-tune) • (b) Use case - Quantization could affect a use case like code generation differently than function calling, see hypothetical diagrams below

Generally, the goal with quantization is to hit a sweet spot on the Pareto curve of quality vs speed. We work with enterprise customers individually to find this spot. However, for our public endpoints there’s no one perfect configuration since there are a variety of use cases on the platform. KL Divergence - Evaluating Quantization Quality

How should one measure model quality for quantization? Since quality is use case dependent, developers are ultimately the best judges for quality for their application. However, to measure general model quality, we prefer to focus on divergence metrics (how much the quantization changes outputs of a particular model) as opposed to pure capability metrics (i.e. how much the quantized model scores on general benchmarks like MMLU). This idea is well described in a recent “Accuracy is Not All you Need ” paper from Microsoft Research. Simply put, quantization introduces noise that can flip some right answers to wrong but also some wrong answers to right (especially when the model is “on the fence”). This skews accuracy. Focusing on changes in model probability distribution is more precise and hence gives enough resolution to interpret effects of individual quantization techniques. Specifically we focused on two divergence metrics: • Kullback-Leibler Divergence (KLD) - measures how much token probability distribution changes (even if the selected token in each position is still the same) • Token rejection rate - measures how many selected top-probability tokens differ (you can think of it as an accuracy of using the quantized model as a draft model (β) )

We further breakdown these metrics for prefill and generation (different parts of inference may use different quantization techniques) to understand the divergence: • Prefill KL Divergence • Generation KL Divergence • Prefill Rejection Rate • Generation Rejection Rate

Our methodology is as follows: Reference model generation: To compute divergences, we ask the reference 16-bit model to generate tokens until completion across a varied set of prompts. Assuming sufficient token volume, divergence metrics are robust to the choice of the initial prompts. Reference distribution creation: At every position, we record the top N logprobs and normalize it to a distribution. As a rule of thumb, we pick N such that the top N tokens cover 0.99 of the distribution, and we find N=16 to be a good value. “Forced” quantized model generation and distribution creation: We then run a quantized model over the same prompts and construct a similar distribution. Importantly, we stick to the completions sampled from the reference model even if the quantized model prefers to choose other tokens. It guards against the quantized model generating an entirely different completion that may have poor quality but good perplexity metrics (e.g. repetitiveness). Divergence analysis: We can then analyze the expected divergence across all samples, expected divergence with respect to positions, etc. Importantly, we can look at these metrics for both the prefill, as well as generation, since different techniques affect each section differently.

We evaluated KL divergence on Llama 3.1 8B...