Ai Model Performance Metrics Explained

AI Model Performance Metrics Explained Announcing our Series F . Learn more

AI engineering

AI Model Performance Metrics Explained

Dive into the key model performance metrics for inference and when they matter most.

Authors

Kenzie Amack

Last updated January 16, 2026

Share

TL;DR Users no longer tolerate slow AI. Optimizing performance isn’t as simple as chasing big benchmark numbers. This blog dives into the three key inference metrics that shape perceived performance and shows how different workloads demand different tradeoffs. The “fastest” model depends entirely on how your users interact with your product.

In AI, yesterday’s breakthrough is today’s baseline. When ChatGPT launched in 2022, users were blown away when they received any reasonable response. Just a few years later, and the shiny newness has worn off, along with their originally low expectations. End users can now be found complaining about LLMs being “ frustratingly slow ”. At Baseten, we’ve seen developers focus shift to match these new user expectations. A year ago, very few developers were optimizing for performance, it was all about quality . Now as quality becomes more widely available with recent model releases, the focus has shifted to performance. Developers now come to us because shaving even milliseconds off inference latency can be a step function change in the user experience. As the adoption of AI has increased, end user expectations for performance have quickly risen, demanding the application layer (and their developers) to keep up. But improving performance can mean many different things. It’s vital that developers understand what AI model performance metrics matter, and how they can make thoughtful decisions when optimizing performance with their end users in mind. The basics of performance So, what is “performance” during inference? There are three key AI model performance metrics that heavily influence the end user experience. 1) Time to first token, 2) Tokens per second, 3) End-to-end latency. Here’s what they mean: Time to first token (TTFT): the length of time between when a prompt is submitted and the first response token appears. Lower TTFT is better.

Tokens per second (TPS): how many output tokens are produced per second. Higher TPS is better.

End-to-end latency: the total time from when the prompt is sent to when the final output token is produced. Lower E2E latency is better.

Every user interaction is influenced by these model performance metrics. TTFT influences the user's first impression of a model’s response time since it’s the time a user waits to see the first output token. A low wait time means the user before they know something is working via a partial response quickly and the application is responsive. TPS on the other hand, influences the length of time a user waits to see the subsequent output tokens. A high TPS also means the application can handle many concurrent users, making it robust to requests at scale — an important consideration for LLM inference performance in production environments. These metrics will vary by both API provider and model. API providers have different performance optimizations to make models more performant such as batching, quantization, KV cache aware routing, etc. Performance can vary by model as well. Smaller models are on average more performant because they require less memory bandwidth and compute, allowing higher throughput and lower inference latency on the same hardware. While smaller models are generally more performant and efficient, many developers shy away from them due to the loss of quality that can come with having fewer parameters. However, with techniques like SFT / RL we can push these models to be on par with larger models on quality while retaining their performance benefits. Models also have “personalities” that influence both the output style and the number of tokens they generate for a given prompt. It’s key to understand that latency, TPS, and end-to-end latency will not translate across models. Some models are more “talkative” and create more output tokens on average. It’s key to re-evaluate performance baselines during model shifts. Evaluating API providers When evaluating providers and you’ll want to check for: Does this API provider have consistently good performance across other models I may be interested in? (in case model swapping is needed in the future)

Is performance stable and consistent? Does this provider have good performance all the time, or does it vary in ways that affect my workload?

Is the provider reliable with high uptimes?

Does this provider enable me to scale (higher rate limits or more GPUs) to support my short term traffic?

Developers often struggle to find consistent sources of truth for what “good” performance should look like by model and by API providers. Openrouter and Artificial Analysis can be great places to start LLM performance benchmarks. While these benchmarks work great as developing a baseline, developers should always be wary of “vanilla” benchmarks. Real performance is often achieved by tuning settings to a specific workload. These public benchmarks are a one size fits all approach and likely do not represent the actual performance for each unique workload. While benchmarks are a great way to start, API providers should be able to “tune” models to fit your needs within dedicated deployments at scale. This can create meaningful performance gains above and beyond optimized off-the-shelf multi-tenant APIs. So, what actually matters After every new model drop, the Twitter sphere lights up with new performance numbers for API providers vying to be “the fastest” (for every inference provider, the answer is always - us!). In 2025, the broader AI market was nearly fanatical about TPS (see one Twitter frenzy here ). It’s easy to get worked up about big performance numbers; however, it's key to double click into what these AI model performance metrics actually mean for the end user experience. Let’s take the tweet about TPS from the gpt-oss release (see exhibit below). If you were optimizing for TPS only you would likely pick Provider 3 at ~3,800 TPS. However, if you look at that same providers TTFT they don’t start producing tokens for over 3 seconds! Provider 2 on the other hand which has a TTFT of (0.48) seconds would have already produced 2.3k output tokens before provider 3 generates a single output token. ✕ This is an illustrative snip of Openrouter taken from...