Scaling Agentic Evaluation Swe Bench

Agentic Evaluation: Lessons from 200,000 SWE-bench Runs

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Agentic benchmarks can have hundreds of problems, each requiring dozens of steps. If you want statistical confidence you need to run them multiple times. That adds up to tens of thousands of runs, but evaluation methods designed for short, linear executions don’t hold up once agent trajectories become long, branching, and stateful.

In the process of applying AI21 Maestro , our test-time compute agentic framework, to SWE-bench Verified, we ran over 200,000 evaluations . In this blog, we describe the evaluation architecture we built to address these challenges at scale, and share practical takeaways for developers, AgentOps, and MLOps teams evaluating agentic systems in production.

Read more about orchestrated Test-Time Compute for long-horizon agentic tasks

Why agentic evaluation is an infrastructure problem

Evaluating a standard LLM is already non-trivial, but agentic systems introduce layers of complexity that traditional one-shot evaluation pipelines aren’t architected to handle. Because agents like AI21 Maestro are stateful, active participants—traversing codebases, executing terminal commands, and iterating—the challenges are less about methodology and more about infrastructure bottlenecks .

To reach statistical confidence at scale, we had to solve for three specific technical constraints:

The throughput wall (Duration & Duplicity): Unlike a chatbot that responds in milliseconds, an agentic run is a high-latency, multi-step workflow. When you factor in Duplicity —the requirement to run the same benchmark instance across multiple random seeds to measure variance—you face a massive compute overhead. We weren’t just running a test; we were orchestrating thousands of concurrent, multi-minute stateful sessions.

State collision (The isolation requirement): Agents have the ability to write – they modify file systems and change environment states. To ensure evaluation integrity, every trajectory must be strictly isolated to prevent one agent’s hallucinated code change from poisoning the environment for the next. This creates a massive container orchestration challenge: how do you provision thousands of pristine sandboxes without the overhead killing your performance?

The reliability tax (Resumability): In a campaign of 200,000 runs, infrastructure failures (pod evictions, network blips, or runner timeouts) are a statistical certainty. In long-running evaluations, losing a 2-hour trajectory at the 99% mark is a massive waste of tokens and compute. We needed a decoupled architecture that separates Generation (patch creation) from Evaluation (test execution), allowing for granular resumability.

These aren’t just theoretical hurdles; they represent the primary technical friction that caused our initial local and naive Kubernetes implementations to fail under load.

Trial #1: The “local-first” assumption meets cloud reality

The official SWE-bench code was written to run locally. It assumes you have a Hugging Face cache directory to avoid re-downloading the benchmark, and it spins up Docker containers both for inference and for evaluation, assuming docker run just works.

In practice, meaningful evaluation doesn’t happen on a local machine beyond the initial debugging phase. While local runs are useful for quick iteration and setting breakpoints, evaluation at scale requires proper infrastructure. Ours is Kubernetes-based, orchestrated via Argo Workflows – an environment where we can reliably measure performance, quality, and the white-box metrics we care about, while benefiting from elasticity, fault tolerance, and horizontal scaling via the cluster’s HPA.

Unfortunately, none of the SWE-bench assumptions holds true in Kubernetes.

First, fresh pods don’t store cached data. Simply running the SWE-bench code as is resulted in each pod attempting to download the benchmark from HuggingFace on startup. At scale, we started getting hit with 429 rate limits – not because of any sophisticated attack, just because we were accidentally requesting the same files thousands of times.

Second, it was clear that just running SWE-bench code as-is in our Kubernetes environment won’t work. You can’t simply docker run inside a Kubernetes pod and expect it to work smoothly. Container-in-container execution requires privileged access, adds overhead, and complicates resource management. We needed to modify the way we were running the code to make it compatible with our test environment.

Trial #2: Adapt the SWE-bench code

As a second attempt, we tried to keep SWE-bench’s original Docker-based evaluation logic and make it work inside our Kubernetes environment. We adapted the code so that each evaluation run would provision a fresh Docker container from within the Kubernetes pod running the evaluation, download the repository, install dependencies, execute AI21 Maestro, evaluate the resulting patch, and then tear everything down when finished.

At the scale of SWE-bench this quickly adds up. With roughly 500 instances, running two variants and repeating each run multiple times to measure variance (e.g, duplicity 4, 8,16) meant provisioning and tearing down on the order of 16k Docker containers over a single evaluation window.

It ran – but slowly and with significant failure rates caused by resource contention and external rate limits.

We had to find a way to declutter the system and improve uptime.

Trial #3: Sharing resources across runs

Our real ‘aha!’ moment came when we tallied what was actually unique per run, versus what could be shared across all runs for that instance:

What if we could provision what was shared between runs – and then allow many subsequent runs to use those resources? In other words, what if we could make the simulation environment multi-tenant?

So, we streamlined our evaluation stack to resemble this architecture:

To test this new multi-tenant simulation environment, we deployed approximately 500 pods in our Kubernetes cluster (one for each SWE-bench instance). Each pod contains the shared resources listed above:

The repository checked out at the correct commit

The MCP server ready to execute commands

Installed dependencies that persist across runs

These pods are provisioned just once (at the evaluation’s start) and can...