Firefunction V2 Launch Post

Firefunction-v2: Function calling capability on par with GPT4o at 2.5x the speed and 10% of the cost=

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Firefunction V2 Launch Post Firefunction-v2: Function calling capability on par with GPT4o at 2.5x the speed and 10% of the cost=

PUBLISHED 6/17/2024

Table of Contents Highlights Intro The state of function calling Firefunction-v2 creation process Evaluation Highlighted capabilities Getting Started Conclusion Acknowledgements

Table of Contents

Highlights

• We are releasing Firefunction-v2 - an open weights function calling model. Quick links: Docs , model playground , demo UI app and Hugging Face model page • Firefunction-v2 is optimized for real world scenarios including multi-turn conversation, instruction following and parallel function calling. It retains Llama 3’s multi-turn instruction capability (0.84 vs 0.89 on MT bench) while consistently outscoring Llama 3 on function calling tasks (0.51 vs 0.30 on Nexus parallel multi function eval) • Firefunction v2 is competitive with GPT-4o function calling capabilities, scoring 0.81 on a medley public benchmarks vs 0.80 for GPT-4o. It’s available at a fraction of the cost of GPT-4o ($0.9 per output token vs $15) and with better latency (180 tok/sec vs 69 tok/sec).

Intro

Over the last few months, we’ve seen a huge leap in large language model capabilities with model releases like Llama 3. One area where improved model reasoning ability is particularly valuable is function calling. We’re excited to build upon recent model advances and push the frontier of function calling with the release of our latest function calling model, Firefunction-v2. Firefunction-v2 is designed to have the intelligence and generalizability to handle real-world agentic use cases, like chat assistants, which involve function calling capabilities, alongside general chat and instruction following. Highlights include: Llama-3-esque chat capabilities: Preservation of chat and other generalized capabilities from Llama-3 70B (Firefunction-v2’s base model) Better function calling, especially for multiple functions: Better ability to handle more complex function calls and follow instructions for function calls Faster and less expensive than GPT-4o - Enjoy similar function calling ability to GPT-4o with better UX and lower costs. Reach speeds of 180 token/sec and pricing of $0.9 per 1M tokens, instead of speeds of ~69 tokens/sec and costs of $15 per 1M output tokens

Firefunction-v1 Firefunction-v2 GPT-4o Single turn function call (routing) ✅ ✅ ✅ Multi-turn conversations 😐 (limited) ✅ ✅ Parallel function calling ❌ ✅ ✅ Instruction following 😐 ✅ ✅ General conversation (with optional function calling) 😐 ✅ ✅ Cost per 1M tokens $0.5 $0.9 $5 (input), $15 (output) Response latency Up to 200 tokens/sec ~180 tokens/sec ~69 tokens/sec Combined benchmark scores (MT bench, Gorilla, Nexus) 0.49 0.81 0.80

The state of function calling

Function calling - 1 year in retrospect It’s now been almost exactly 1 year since OpenAI unveiled function calling as a feature . Function calling is the ability for a model to output structured text to call external APIs. LLMs have immense utility on their own but cannot access real-time or internal data. Function calling bridges that gap by letting LLMs format text to call APIs and optionally incorporate the API response. In the last year, we’ve seen a proliferation of function calling and agentic usage. Fireworks was one of the first to offer a function calling-focused improvements in our models by launching: (1) Firefunction-v1 and Firefunction-v0 - Firefunction-v1 excels at structured output and simple function calling use cases. It powers a variety of routing and decision-making use cases for developers today. (2) Structured output modes - Options to guarantee that any of Fireworks’ LLMs adhere to a desired output format, like JSON Most major closed-source models, like Google’s Gemini and Anthropic’s Claude, have added function calling capabilities and several open-source function calling-models have been released. While function calling has massive potential, there’s been limited success productionizing the capability. Developers have to choose between different tradeoffs with open-source vs closed-source models. Traditional open-source function calling models - overly specialized Most open-source models have focused on narrow use cases. They introduce function calling capabilities through an aggressive fine-tuning process that focuses narrowly on function calling and erases the native capabilities of the base model. This creates models that perform well on benchmarks (usually emphasizing single-turn, forced function calling). However these models (1) Don’t generalize outside of benchmarks to real world scenarios and (2) Are poor at general reasoning and chat. Closed function calling models - slow and expensive for production usage Given the narrow focus of open-source function calling models, closed source function calling models like GPT4 and Claude tend to significantly outperform open-source models in non-function calling tasks, like chat and text generation. However, these models have both high latencies and costs, limiting their production usability. Moreover, generalized models weren’t specifically designed for function calling and have limited ability in structuring responses and determining when to call functions. Firefunction-v2 creation process

One counter-intuitive thing we’ve learned is that real-world function calling models must be great at non-function calling tasks. When Llama 3 and Firefunction-v1 were released, we received a flood of user comments like: • “Llama 3 is insanely good. Can you just add better function calling to it?" • “Is [your function calling model] useful for chat conversations or only useful for function calling?” • “I’m using Llama 3 for function calling in my app right now. It mostly just chats with users but every once in a while, we need it to call an API”

This feedback led us to depart from the aggressive, narrow fine-tuning approach used for other open-source function calling models. Instead of overfitting to function calling scenarios, we augmented the base model with function calling capabilities while preserving its instruction following abilities. We selected llama3-70b-instruct as the base for Firefunction v2, based on its excellent performance in real world...