Tool Calling In Inference

Tool Calling in Inference Announcing our Series F . Learn more

AI engineering

Tool Calling in Inference

Dive into the basics of tool calling, why tool calling quality fluctuates between providers, and how Baseten builds reliable and scalable tool calling for LLMs.

Authors

Kenzie Amack

Bryce Dubayah

Last updated November 4, 2025

Share

TL;DR The rise of AI agents has fueled a surge in open-source models that support tool calling, but developers have quickly realized that the quality of tool calling varies among inference providers. Inference providers play a critical role in ensuring tool calling success from pre-processing to model execution, all the way to post-processing. This post breaks down the tool calling basics, how to find the best agentic model, and unpacks what inference providers can do at each layer to ensure reliable, high-quality inference for agentic workloads.

Over the past year, the rise of AI agents has fueled an explosion of open-source models that support tool calling. If you’re a consumer of AI trends, you may have also noticed the explosion of something else around that time: tool calling benchmarks. While benchmarks have always been the darling of the AI world, the consumption of third-party benchmarks related to tool calling seems to take on a whole new fervor. Developers have caught on that there is a significant range in tool calling success between different inference providers, and they’re grasping for answers on who does it best. Historically, inference providers had relatively similar model quality (quantization fixed). The tool calls upend this historical trend and introduces a particularly opaque criterion for developers to evaluate. As the ecosystem races to benchmark and compare performance, it’s worth looking beyond what’s being touted in the Twitter sphere to make your own informed decision. In this blog, we’ll cover everything you’ll need to make informed decisions when evaluating tool calling success among providers. We’ll unpack what these benchmarks evaluate, how inference providers influence tool calling outcomes, and how Baseten is working to deliver best-in-class tool calling for developers. Tool calling demystified At its simplest, tool calling is how LLMs interact with external applications. Through tool calls, models retrieve, analyze, and generate information. LLMs began using external functions with function calling in 2021 and can now utilize multiple tools and further orchestrate external applications. By offloading certain tasks to tools, models can remain relevant for longer periods without requiring retraining. Tool calling also increases efficiency (less needs to be stored in model weights), ensuring models can be more dynamic to adapt to user requests. ✕ Agentic workflow overview For example, ChatGPT can query your contacts within your Google Suite for an email or phone number (contact lookup) or search and analyze documents in Slack (internal knowledge search). Both of these are tool calls that enable models to utilize user context to create personalized product experiences. When coordinating with external applications, models must generate specifically formatted text (a schema) that specifies which tool to “call” and what inputs must be provided. For the contact lookup, ChatGPT may invoke a contact lookup tool that requires  { "name": "Jane Doe" } as the input to return the email address for Jane. Tool calls can be single-turn and multi-turn. In a single-turn tool call, the model requests a single tool (indeed, it’s aptly named) and returns the result. But the real fun starts with multi-turn calls. Developers utilize multi-turn for complicated requests where multiple different applications must be called. Let’s take an agentic workflow that generates an outbound sales email. One tool generates a prospect that matches the title the user supplied, the next tool generates recent company news to include, and the last tool combines these inputs to create an email template. Each tool is kicked off in a chain-like pattern where the output of one tool becomes the input for the next. While multi-turn tool calls are powerful, output quality can degrade with each successive “turn”. Each turn requires the model to interpret the received output correctly and translate it into a schema that the next tool will accept. All of this translation creates room for error. In addition to single-turn and multi-turn tool calls, there are four types of tool choice implementations:

How to select the right model and provider Benchmarking models 2025 is the year of agentic AI for a reason. Since the release of DeepSeek R1 in January 2025, there has been a significant rise in models that can power agentic workflows. Thankfully, you now have more models than ever to choose from, but it’s still important to ensure you’re utilizing the model best fit for your agentic use case. When testing with your prompt, you’ll want to monitor model outputs across: Tool selection accuracy: Did the model pick the right tool(s), in the right order?

Argument fidelity: Were inputs (into the tool call) complete and grounded in context?

Schema validity: Did the outputs match the prompted schema and return valid JSON?

Turn efficiency: how many calls and tokens did it take to complete each task?

Thankfully, you don’t have to start from scratch. Libraries like BFCL , ToolBench , and ShortcutsBench can serve as a great starting point to show results across the dimensions. While it may be tempting to opt for the highest-rated model and run with it, we suggest testing your workflows against multiple LLMs. Each LLM has a different “personality” and may be uniquely powerful for your workflow (regardless of public benchmarking scores). Benchmarking inference providers Once you’ve selected your model, you’ll want to test the same prompt across multiple inference providers to choose the provider that gives you high tool calling accuracy, reliability, and the right mix of latency and throughput. Buyer beware: it’s tempting to skip this step after all the work of finding the right model. But tool call success is very reliant on having the right inference provider. Public benchmarks have shown some of the worst providers give a success rate of only 7% - meaning for every 100 tool call attempts you make, only seven go through. Sometimes, you get what you pay for. To avoid a situation like the one above, it’s crucial to benchmark inference providers. There are plenty of...