Fireoptimizer

FireOptimizer: Customizing latency and quality for your production inference workload

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Fireoptimizer FireOptimizer: Customizing latency and quality for your production inference workload

PUBLISHED 8/30/2024

Table of Contents

Why adaptation matters in AI inference

Adaptive Speculative Execution with FireOptimizer

Automated Improvements with Adaptive Speculative Execution

Customer Stories

Data Privacy and Security with FireOptimizer

Get Started with FireOptimizer

Table of Contents

At Fireworks, we've always believed that off-the-shelf models need to be adapted to meet production-grade performance. Today, we’re excited to introduce FireOptimizer , our adaptation engine designed to customize and enhance AI model performance for your unique use cases and workload. We have launched a new FireOptimizer feature: adaptive speculative execution , which delivers up to 3x latency improvements by tailoring speculative execution to your specific data and needs automatically. Why It Matters: In today’s world, every millisecond counts. Whether you’re powering real-time customer interactions, processing large-scale data for intelligent search, or using AI to generate code, FireOptimizer simplifies the complex tuning work for optimizing latency and quality, and ensures your models are not just fast, but customized to perform at their best for your unique scenario. The Benefits: • Faster Inference: With FireOptimizer’s adaptive speculative execution, production workloads across various models saw up to 3x latency improvements, ensuring applications are always highly responsive. • Hassle-Free Optimization: FireOptimizer automates the complex optimization process, so you can focus on building your application, while we handle the rest.

Why adaptation matters in AI inference

Many developers are surprised by the extent that results can vary serving the same model on the same hardware. For example , Llama 70B on eight GPUs in a volume-optimized set-up can be 4x cheaper per token than Llama 70B on the same eight GPUS optimized for single request speed. This variation occurs because text distributions and inference requirements may vary dramatically, and a one-size-fits-all approach often fails to deliver optimal performance. Adaptation is necessary because multiple parts of the serving stack can be adjusted based on specific use cases, enabling organizations to maximize efficiency, quality, and cost-effectiveness. It involves a holistic approach that considers every layer of the deployment stack—hardware, model, and software—to ensure that all components are optimized to work together seamlessly. Adaptation is especially critical when building compound AI systems, where multiple models and processes interact, requiring precise alignment and optimization across the entire system to achieve peak performance. FireOptimizer allows users to customize their LLM inference set-up for ideal cost, quality and performance. FireOptimizer enables improvements across three key layers of the inference stack: Hardware - Which accelerators are you using and how is workload distributed across them? For example, on the same 8 GPUs , you could host 8 separate copies of the model or you host one model copy across multiple GPUs. Model - How is a model quantized , tuned, or otherwise customized? Software - How are requests processed, cached, and handled? For example, how are prompts vs generations processed?

FireOptimizer employs automatic capabilities to enable adaptation as well as extension to the traditional inference performance improvement approach. Some of these techniques include: • Adaptive speculative execution • Adaptive caching • Customizable quantization • Adaptive fine-tuning • Personalized fine-tuning at scale • Customizable hardware mapping

We have already shared customizable quantization based on quality and latency tradeoff curve before. In this post, we’ll focus specifically on adaptive speculative execution. Stay tuned for more deep dives and future releases. Adaptive Speculative Execution with FireOptimizer

FireOptimizer employs a new technique called adaptive speculative execution which improves performance by tailoring speculative decoding to your specific workload. For context, speculative decoding is a technique that parallelizes token generation to accelerate inference. Typically, a large language model (LLM) generates tokens one by one, but speculative decoding changes this by using smaller "draft" models to predict possible token sequences in parallel with the main LLM. The main LLM then verifies these predictions—if correct, the draft model's output is used; if incorrect, it’s discarded.

Speculative decoding works based on the two key ideas of : • Speed of Verification vs. Generation: It’s faster for the LLM to verify a sequence of predicted tokens in parallel than to generate each token individually. • Hit Rate of the Draft Model: The effectiveness of speculative decoding depends on the draft model’s accuracy, or "hit rate." A high hit rate allows the LLM to incorporate more correct predictions, boosting speed without sacrificing quality.

This approach can significantly reduce latency without compromising the response quality, as the LLM ensures the final output remains accurate. However, speculative decoding that uses generic draft models has limitations. Why Generic Draft Models Fall Short: In traditional speculative decoding, draft models are typically trained on generic data, like publicly available conversation datasets. While this can work for general use cases, the hit rate—and thus efficiency—declines significantly when applied to more specialized scenarios. Adaptive Speculative Execution: FireOptimizer improves upon this technique, unlocking greater potential with ease. FireOptimizer’s adaptive speculative execution is built on two key concepts: Profile-driven Customization: FireOptimizer enhances the "hit rate" of the draft model by leveraging the inference workload profile. The higher the hit rate, the greater the latency improvements you’ll experience. Automatic Training and Deployment: FireOptimizer handles the training and evaluation of the draft model for you. Simply provide your data, deploy the resulting draft model, and enjoy significant latency reductions without the hassle of manual tuning.

The advantages of this technique are: • Broad...