Fireworks Ai Fast Affordable Customizable Gen Ai Platform

Fireworks.ai: Fast, Affordable, Customizable Gen AI Platform

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Fireworks AI Fast Affordable Customizable Gen AI Platform Fireworks.ai: Fast, Affordable, Customizable Gen AI Platform

PUBLISHED 8/17/2023

Table of Contents

Developer-Centric AI

Fast, Affordable LLM Inference with Fireworks

Special Serving & Performance of LoRA Tuned Models

Simple Workflow for Experimentation and Production

Your Go-To Spot for Fine Tuned Models

Customization with Fireworks Fine Tuning Cookbooks

Conclusion

Table of Contents

tl;dr Fireworks.ai releases the fast, affordable, and customizable Fireworks GenAI Platform . It enables product developers to run, fine-tune, and share Large Language Models (LLMs) to best solve your product problems. The platform provides state-of-the-art machine performance for latency-optimized and throughput-optimized settings, cost reduction (up to 20–120x lower) for affordable serving, and a customizable cookbook for tuning models for your product use cases. Developer-Centric AI

Generative AI (GenAI) has shifted the product landscape and redefined product experiences for consumer and business-facing products. Large Language Models (LLMs) enable never-before-seen performance on tasks like document/code generation, auto-completion, chat, summarization, reranking, and retrieval-augmented generation. Foundation models (FMs) and parameter-efficient fine-tuning (PEFT) now enable more efficient AI customization. Instead of training large models from scratch with vast data, companies can now tailor GenAI models using open FMs like LLaMA, Falcon, and StarCoder. These FMs, sourced from internet data, serve broad tasks. But with PEFT techniques like LoRA , FMs can be customized for areas like legal or finance. Models can be customized with company data for specific tasks or personalized AI products. This approach speeds up product development and reduces data needs.

Open LLMs and PEFT enable customization of foundational LLMs for new use cases At Fireworks, we believe in developer-centric AI : LLMs should be fast, affordable, and customizable for integration into modern products. Fireworks provides: • A fast and affordable inference service that you can seamlessly integrate with your products with a simple API call. • An open collection of foundational and fine-tuned models so that the community can share and easily experiment with customized LLMs without having to figure out deployment details. • A simple and easy-to-use cookbook for fine-tuning LLMs and the ability to upload these customized models to our inference service . We aim to empower developers to integrate new AI building blocks into new and innovative products.

Fast, Affordable LLM Inference with Fireworks

Efficient Inference

Efficient inference of LLMs is an active area of research, but we are industry veterans from the PyTorch team specializing in performance optimization. We use model optimizations including multi/group query attention optimizations, sharding, quantization, kernel optimizations, CUDA graphs, and custom cross-GPU communication primitives. At the service level, we employ continuous batching, paged attention, prefill disaggregation, and pipelining to maximize throughput and reduce latency. We carefully tune deployment parameters including the level of parallelism and hardware choice for each model. Machine efficiency improvements from our runtime allow us to pass on cost savings to the user. Compared to other providers serving similar-sized models, we offer prices that are 1–2 orders of magnitude lower. In particular, we provide special optimization for fine-tuned models, resulting in significant savings compared with OpenAI and other OSS model providers.

The Fireworks GenAI Platform delivers significantly lower costs than comparable providers The Fireworks Developer tier is free to use to get you going easily. For more advanced and much higher usage, Fireworks Developer Pro pricing is based on $/input million tokens and $/output million tokens, listed in the table below. To illustrate the cost for product usage, the below study examines several common LLM use cases and the typical number of input and output tokens. We use the Fireworks per-token pricing and compute a normalized price for each use case.

In addition to cost savings, we can deliver lower latency and cost than existing solutions. For example, we compare with the popular open-source vLLM framework. Running both solutions on a lightly loaded server with 8 A100 GPUs we're getting 2–3x lower latency. Importantly, our service's latency stays low as server load increases and it can maintain 3x higher maximum throughput than vLLM while staying within the latency constraint.

The Fireworks platform offers latency significantly lower than comparable open-source offerings Special Serving & Performance of LoRA Tuned Models

The Fireworks Inference Service stack is built with first-class support for serving of LoRA fine-tuned models. We compose LoRA with all optimization techniques, including sharding and continuous batching. Further, we enable multi-tenancy of multiple LoRA adapters on the same base model with cross-model request batching. This brings efficiency that is not available when hosting your tuned model elsewhere. We pass these savings to you by making the cost of a fine-tuned model serving as low as the base model regardless of how much traffic it receives. We believe this pricing is key for unlocking the power of expert models fine-tuned for a specific use case. If you have many use cases or many variants of a single model, each of the model variants might serve very few requests. Deploying each of them on a separate GPU would be prohibitively expensive on a per-token basis. With Fireworks, you can experiment with fine-tuning without breaking the bank.

The cost of serving fine-tuned models on Fireworks does not scale with the number of models Simple Workflow for Experimentation and Production

Simple Interfaces and APIs

Fireworks.ai provides several state-of-the-art foundational models for you to use off the shelf, including those from the LLaMA 2 , Falcon , and StarCoder families. Sign up for an API key here and access the models for free today. The Fireworks console allows you to interact with models right in your browser .

Trying out LLaMAv2–70B in the Fireworks console Programmatic API Access...