Mind The Gap

Mind the gap: Separating demo agents from production systems

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

TL;DR

AI in production is won or lost in the gaps between a promising model run and a production AI system that can reliably validate, route, orchestrate, and decompose work under real-world constraints.

The four gaps between model demos and production AI systems

When I sat down with Barak Lenz, our CTO here at AI21 for a recent episode of YAAP , I tried to do what every host does. I asked him for the TL;DR. I wanted the one liner that would hook the audience. His response was very typical of him.

“I don’t really want to start with a one liner,” he told me. “I don’t like one-liners. They tend to throw away the specific details that actually explain what’s going on.”

It wasn’t surprising if you know him, but a refreshing moment in the AI landscape. In a world of hype, Barak isn’t interested in research papers that look good on Twitter. He is interested in systems that actually run on time, within budget, and without failing silently after thirty minutes of compute.

Our conversation centered on a single, provocative thesis. The industry is looking at average scores on leaderboards, the community is obsessed with yet another OpenClaw version. but production is won or lost in the Gaps. To build what Barak calls an AI Operating System, we have to stop looking at LLMs or Agents as magic boxes and start looking at the distance between current performance and theoretical potential. Here is my breakdown of the four gaps Barak identified, and why they change everything for those of us building in this space.

1. The Validation Gap (The “Oracle” problem)

Barak began by pointing out a gap that is well-known in the literature but rarely solved in production: the Oracle Gap. This is the mathematical difference between Success@1 (getting it right on the first try) and Success@N (getting it right at least once across several attempts).

If you run a model once, you are simply pulling a result from a distribution. But if you run it N times and pick the best one, success rates on reasoning benchmarks often spike from the 70s into the 90s. The gap exists because the system doesn’t always know which of its own answers is correct.

The principle is almost a law: more spend leads to more quality. But it’s intuitive, not trivial. Simply spending more with stronger models or multiple runs doesn’t necessarily work, because the system still has to identify the best answer. You can see it clearly in the chart below: the Oracle line climbs steadily as ensemble size grows, while the Average flatlines, and the Adversary collapses.

Figure 1: Oracle vs. Adversary performance (from this paper ) across ensemble sizes (N) for Gemini 2.5 Pro, Claude 3.7 Sonnet, GPT-4.1, and a Mixture approach. The Oracle line – representing the best possible pick from N runs – climbs steadily toward 75–80%, while the Adversary (worst pick) collapses toward 25%. The Average flatlines. The gap between Oracle and Average is exactly what better validation could recover.

This isn’t just about “agents” guessing a reasoning path. I’ve seen this exact behavior in the way we handle data retrieval. In our recent research at AI21 on multi-scale indexing , we applied this “Success@N” logic to RAG. Most systems commit to a single chunk size upfront, but we found that different queries fundamentally require different resolutions.

By indexing the same corpus at multiple sizes and using a principled way to aggregate them, we were essentially trying to close that same “Validation Gap.” When we ran “Oracle” experiments – choosing the optimal chunk size per query – we found 20-40% headroom left on the table by traditional, single-path systems. Barak’s point is that we shouldn’t settle for the performance of a single rollout when a verified, multi-path approach is within reach.

2. The Contextualization Gap

As Barak and I moved deeper into the architecture, I began to see these gaps not as isolated hurdles, but as a logical progression. The Validation Gap we discussed is essentially the delta between a model and repeated calls to that same model. But what if we could choose entirely different models for different inputs?

This is where the Contextualization Gap comes in. Barak’s core insight here is that leaderboards are fundamentally misleading because they report what is best on average, while real systems operate on specific, wildly varying inputs.

“This gap is the difference between what’s best on average and what’s best for a specific input.”

Figure 2: Each column is a benchmark problem; green means a model solved it. Notice how GPT-5 Mini (56%) and the frontier models (70–76%) share a massive overlapping green block on the left – the “easy” problems every model gets right. The expensive flagship models earn their keep only on the harder right-hand side. Routing the easy problems to the cheaper model wouldn’t cost you much quality – but it would save you a fortune.

When you look at benchmarks input-by-input, a pattern emerges: easier problems are often solved by much cheaper, smaller models with almost perfect overlap with the flagship ones. In practice, this means a massive fraction of your compute budget is likely being wasted on “overkill” models for tasks that never required them.

We can see the data for this in the way LMArena has always maintained a division of different tasks and specific category scoring alongside overall rankings. The “best” model overall isn’t always the best – or the most necessary – for every specific sub-task.

To me, the missing piece in the current conversation is a clear presentation of where this gap comes from and what our other options are. It isn’t just about running multiple models or agents on the same input and choosing between them after the fact. Even if we had a perfect validator, that approach is incredibly wasteful.

The real question we should be asking is: What if we could know which path to choose during the run and not after?

The solution Barak proposes is to move away from a monolithic “agent” and toward a Portfolio of Actions. This requires an “Action Model”: an execution strategy that...