Test Time Compute Swe Bench

Test-Time Compute Orchestration for SWE-bench with AI21 Maestro

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

TL;DR

AI21 Maestro is a general-purpose agentic framework that automatically scales compute and optimizes orchestration. We demonstrate how it significantly improves model performance on SWE-bench-verified through structured plans, automatic horizontal scaling and decision-theoretic optimization techniques.

Introduction

As agentic tasks grow more complex and the variance in execution increases, the core challenge becomes how to allocate test-time compute (TTC) efficiently across vast and extended trajectories.

Reasoning models were an early attempt to answer this question. These models dynamically allocate more tokens to “think” and better choose their trajectories. Later, strategies like “best of N” (also using more compute to improve accuracy), or “agent/model routing” (learning to route for the cheapest model that solves the problem) joined the agentic game and showed value in many use cases.

While useful, those techniques suffer from two shortcomings:

They rely on “blackbox orchestration”: Current strategies treat each run as a sealed unit rather than part of a coordinated team. These runs act like individuals in separate rooms, agnostic of each other’s progress and only evaluated post-execution.

They are trained once, and subsequently applied at inference time according to the strategy with the best average performance on the training set. That means that they not only suffer from the divergence between the distribution of the training set data and that at inference time, but also require full retraining for every new model, agent, prompt, or tool that the system can use.

Scaling to more powerful models fails to resolve these fundamental flaws, as the underlying limitations are architectural. We argue that addressing these issues requires structured Test-Time Compute mechanisms , specifically, systems that allocate resources explicitly and adaptively during execution to enhance accuracy under resource constraints.

AI21 Maestro is such a general-purpose agentic framework. It separates decision making about orchestration from the LLM reasoning itself, allowing better control over long-horizon, multi-step tasks. To demonstrate its power, and explain some of its workings, we show its impact on SWE-bench tasks.

What we learned from evaluating agents across 200,000+ SWE-bench runs

Applying AI21 Maestro to SWE-bench

SWE-bench-verified is one of the best known agentic code benchmarks. Its tasks require dozens if not hundreds of steps, which amplify problems such as agents getting stuck or costing a lot to solve a simple problem. Dedicated SWE agents with specific logic and custom tools have dominated the leaderboard for a while, but recent frontier models have caught up and their “naive” execution with a simple bash tool almost matches the tailor-made agents.

To fairly evaluate AI21 Maestro, we took the following approach: Maestro was only given bash access (and LLMs), and was compared against mini-swe-agent, the ReAct-based agent published by SWE-bench, using the same models and bash tool. We specifically tested AI21 Maestro with GPT-5 and GPT-5 mini, but the same method and gains can be shown for other models.

Pink bars are the result of running AI21 Maestro. All the rest are the published results on SWE-bench.

As seen in the above chart, AI21 Maestro significantly improves the performance of GPT-5 and GPT-5-mini. In fact, with Maestro, GPT-5 performs at the general level of Claude 4.5 Opus and Gemini 3 Pro, the top published performers.

To understand where this improvement comes from, we focus on three components of Maestro (there are others, which we’ll cover elsewhere):

Horizontal scaling

Structured plans

Exploring the action space

Component 1: Better cost/accuracy with horizontal scaling

As already shown by a few dedicated SWE agents (e.g., TRAE), horizontally scaling cheaper models may be more cost effective than using a stronger, larger model, provided you can reliably identify the best output. In the chart below, this tradeoff is clearly demonstrated: Running 8 trajectories of gpt-5-mini yields a higher score with lower cost as compared to running gpt-5 once. But two challenges remain: (a) scaling automatically and only when it is valuable; and (b) choosing the right trajectory.

Percent of resolved items and the median cost per item, across 1, 2, 4 and 8 parallel runs of each model, assuming an oracle reducer that chooses the correct answer. Cost is calculated based on the public OpenAI API pricing.

Maestro solves the first by being built with Test-Time-Compute in mind from the ground up. Expected cost and value is modeled into every “branch” that Maestro considers, based on either learned priors from offline simulation, or manual input from the developer. That way, instead of manually coding parallel execution workflows and optimizing among them, the developer can let Maestro parallelize when possible based on budget.

Whenever Maestro branches out, it keeps full observability and control over those trajectories. Validators are run at the end of the branch, and a Reducer mixes/chooses the final candidate from the branches. This means Maestro can stop agents that get stuck, or just terminate all running branches if one branch finished successfully – without the developer writing any custom code or logic to accomplish this. Maestro’s Execution Engine has built-in support for parallelizing even with state-changing actions, whereas naive parallel execution would otherwise cause conflicting writes and inconsistent state. This approach alone brings immense benefits and potential, as the scores show.

Read more about how we solved state changing actions in parallelism

Component 2: The power of structured plans

Most general-purpose agents today are descendants of the ReAct framework: agents built around raw prompting and ReAct-style loops. Planning, decision-making, and execution are mediated through token generation, with choices such as when to continue, retry, branch, or stop being inferred from next-token predictions. The model encodes its entire control flow inside its textual context, which makes the agent look like a program, but behave like a stochastic process with high variance in cost, latency, and accuracy.

The cost of each item in the dataset was measured across 16 runs, and...