Evaluating Coding Agents On The Work Developers Actually Do

The hidden mismatch: Evaluating coding agents on the work developers actually do | AI21

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Back to Blog

-->

In brief

We’ve all heard it before: your AI system – whether an agent, harness, or routing layer – is only as good as the evaluation behind it.

We’ve also heard: public benchmarks are poor proxies for real developer traffic. In terms of what they ask for, how they phrase it, task difficulty, and how trajectories unfold over the session.

We saw this discrepancy up close when we surveyed our internal developer traffic. The results challenged our assumptions about agentic coding tasks, and we used these lessons to build a robust benchmark that preserves quality on real-world coding tasks in our intelligent gateway. This post covers that journey.

The real distribution of developer work

Luckily, to understand the gap between public benchmarks and real usage, we have the benefit of real live data from our internal team of developers. To make sense of this traffic, we developed an internal coding task classifier and ran seven weeks of real developer traffic through it.

What we saw when we looked at the resulting task distribution completely challenged our intuition about agentic coding habits:

FIGURE 1. Top-level categories were derived from the data, and subcategories were derived inside each category. Total of 15 categories, 84 subcategories, and 4,211 turns.

If you look at the tasks measured by a standard benchmark like SWE-bench, it would seem that developers use agents primarily for pure code generation and fixing bugs from a clean slate. But our classifier revealed a totally different reality.

Lessons from seven weeks of real-world developer traffic

Developers use agents as thought partners, not just as code generators. In our traffic, only about 11% of real developer turns ask the agent to write code. The vast majority of the time, developers are asking clarifying questions, pasting context, doing analysis, or requesting reviews. The single most common message we see is a clarifying question.

Interactive workflows vs. single-turn tasks. The data shows that developer-agent interactions are highly conversational. Developers aren’t just dropping a 1,000-word GitHub issue into a chat and walking away. They are typing short, 80-character messages mid-session. They are debugging step-by-step, pasting console logs, and iterating on the agent’s output.

Where real-world developer traffic and public benchmarks diverge

Task type

We ran several public benchmarks (including SWE-bench, HumanEval, and Spider) through this same classifier to build out our comparison. The mismatch with our internal data was glaring. Almost all of these public benchmarks focus on writing code from scratch or fixing a bug to pass a test.

FIGURE 2. The same categories applied to real usage and to 23 benchmarks. Number in cell reflects % of a dataset’s message. Confirmations, redirections, git commands and pasted context make up about 40% of real user turns, yet appear in no coding benchmark.

Despite the fact that confirmations, redirections, git commands, and pasted context make up about 40% of real developer turns, these interactive properties and context-sharing behaviors appear in almost no public coding benchmarks.

Task difficulty

Beyond not covering real types of coding work, public benchmarks also fail to accurately reflect task difficulty . Standard evaluation sets often test deep, architectural knowledge of a codebase’s internals. But in the real world? A developer is much more likely to ask a quick, practical question like, “Why do I not see my changes in git status?”

Because benchmarks over-index on these highly complex, multi-file architectural issues, they give us a skewed sense of what models actually need to do and can therefore make smaller or cheaper models look artificially incapable of handling everyday traffic. You can see this in the comparison below between the median turn count across tasks from real developer sessions versus a frontier model tested on our benchmark:

FIGURE 3. Median turn count across task categories, as compared between real developer sessions and data generated from running Claude Opus 5.5 on our benchmark.

Task context

We found another major difference between real-world and public benchmark traffic: context. A typical benchmark task is presented as a well-written,1,000-character GitHub issue on a clean code checkout.

But real developer messages are messy. Averaging around 80 characters, they arrive right in the middle of an active session, stacked on top of an extensive history of conversation, tool outputs, and half-finished file changes. Over 60% of our internal sessions contain more than one distinct task. If you evaluate an agent solely on how it handles a single, well-scoped prompt from a pristine start, you are ignoring the reality of mid-session context switching.

Building a real-world benchmark

Defining the goal

Proving the discrepancy between public benchmarks and real-world developer traffic was the easy part. Now, we had to build a new benchmark that would resolve the gap.

The ideal evaluation would simply use our own developers’ real workflows. But building a natural task distribution creates an immediate tug-of-war between three competing properties: drawn from real work, context-rich, and capable of being graded automatically.

Real developer sessions meet the first two criteria, but are almost impossible to grade automatically. There is no pre-written test suite or rubric for a developer asking, “Why is my database connection dropping on line 42?” The intent lives entirely in the developer’s head, making it hard to measure if an agent succeeded without human review. On the other hand, public benchmarks are perfectly runnable and easy to grade automatically, but as we saw earlier, they lack relevance to real-world context.

Our goal was to collapse the tradeoff: combine the automated gradability of public benchmark tasks with the messy reality of actual traffic. We built a new benchmark, guided by four principles:

1. Let real traffic set the quotas. Instead of randomly sampling public tasks, we used the insights from our internal classifier to set quotas. If 40% of our real traffic consisted of clarifying questions and codebase analysis, then 40% of our benchmark...