Fireworks Raises The Quality Bar With Function Calling Model And Api Release

Fireworks Raises the Quality Bar with Function Calling Model and API Release

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Fireworks Raises The Quality Bar With Function Calling Model And API Release Fireworks Raises the Quality Bar with Function Calling Model and API Release

PUBLISHED 12/20/2023

Table of Contents

Function calling through a fine-tuned model vs prompt engineering

Using the API

Future Work

Conclusion

Table of Contents

Since rolling out the Fireworks AI platform earlier, we've been delighted to be part of the development journey of thousands of developers. At Fireworks, our vision has been to provide developers with a fast AI platform with the highest-quality models. We've heard a common user request for function calling, so we're thrilled to announce the Alpha launch of the Fireworks function calling model and API, reaching GPT-4 quality. What is function calling and why is it important?

Function calling is the ability for a model to call external APIs. While LLMs are very useful on their own, they struggle in situations such as: Access to real-time data: LLMs sometimes hallucinate and cannot answer questions that require real-time information. For example, it's difficult for LLMs to answer questions like “What's the current temperature in New York?”. Dynamic Agents: Many of our users create LLM-based agents to take action. These agents sometimes require external information to adapt and decide. For example, a scheduling agent may need to know whether it's raining to cancel an appointment.

Function calling enables LLMs to incorporate knowledge from these API calls to address these issues. For example, an LLM with function calling capabilities could call upon a weather information API to get real-time information or provide an agent with the information to act. Function calling challenges While calling a single weather API could be straightforward for an LLM, the function calling problem space quickly becomes multi-faceted with more complex queries. For example, let's take the example of creating an agent to help with web shopping. • Agent description: Conversational agent that a business deploys to help users make purchases • Functions to use: Entire Stripe SDK • Example queries: (1) Fetching the price of a product (2) Buying a product based on customer intent .

The agent will have to grapple with issues, such as: Intent detection — Does the model know when it's necessary to call an API vs relying on its own knowledge? Function quantity — How does the model perform when many functions are provided? In this example, the agent is provided with the Stripe SDK, which has numerous functions. The agent will need to know when and how to call each individual function. Value Formatting — Can the model correctly structure information, especially for complex data types? In this example, many potential queries utilize more nested structures like the recurring field in the create_price stripe API. This field has a dictionary type where child fields can take on only specific enum values. Contextual Information — Can the model utilize conversational context and knowledge, especially in multi-turn or chat contexts? For example, if a user asks: “What's the stock price of the shirt?” and follows up with a question like “How does that compare to the jacket?”, can the model use contextual info to format corresponding API calls?

Our API is based on a fine-tuned CodeLlama-34B model that we've trained specifically to call functions and converse reliably. This means that our function calling model should have capabilities like a typical CodeLlama-34B-Instruct model (able to remember context) but it can also call functions to access external knowledge and chat with users. Function calling through a fine-tuned model vs prompt engineering

We've fine-tuned a model for function calling, but a popular alternative approach is to engineer a function-calling prompt for a non-specialized model. We predominantly validated that this technique lacks in the following ways: Accuracy — Given aforementioned complexities, it can be difficult for non-fine-tuned LLMs to reliably parse context to understand intent and structure function call arguments. Generalization Ability — To be used in production, a function calling LLM must be able to generate calls for APIs that were not seen during training is crucial to its usefulness in production. Fine-tuning allows us to teach the model to generalize over a wide variety of arguments and functions. The resulting model does well on complex APIs such as Google Place APIs, to Stripe APIs or Plaid APIs even though such APIs don't appear in fine-tuning training data. Tool Chaining — Some use cases for function calling require the model to gather context from both conversation & previous function call outputs to perform function calls. This interaction can become fairly complex as the length of the conversation increases. Dealing with various ways function calling intent & normal conversation can intertwine with each other is a challenging ask for prompt engineered models. Intent Detection — It's crucial to find a balance between a trigger-happy agent & reluctant agent. Both ends of the spectrum result in a bad user experience. It's tricky to construct a prompt to convey a balanced message to the agent about function calling.

Quality Evaluation

To assess these claims and test the accuracy and generalizability of function-calling models, we designed two evaluation datasets. Neither of these datasets was included in model training. Single Turn Dataset — This dataset focuses on single-turn requests, where the LLM response is supposed to be a function call. We assess for accuracy of the function call — right function selection, right argument population & correct output structure. Additionally, the number of functions available is > 10. This dataset is derived from VirusTotalBenchmark . Multi Turn Dataset — This dataset tests scenarios where context for function call is spread across multiple turns — for example a conversation with multiple user statements. Our evaluation is judged on accuracy of the final function call, using the same criterion as described above. This dataset is derived from Glaive2 .

We compare our model against some strong baselines. As GPT-3.5 is not strong in function calling generation, we compare...