How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark \| OpenAI

July 29, 2026

Research Publication

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

Loading…

Share

00:00

_A sped-up video of GPT‑5.6 Sol attempting to solve puzzles in the ARC-AGI-3 benchmark, with the official harness (left) and our Responses API harness (right), which retains reasoning and enables compaction. On the leaderboard for_ _this game_⁠(opens in a new window) _, no frontier model solves any level beyond the first. With our harness, GPT‑5.6 Sol solves all six._

When we first saw GPT‑5.6 Sol’s low scores on the ARC-AGI-3⁠(opens in a new window) benchmark, we were puzzled.

GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture⁠(opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.

Were 2D puzzle games unusually difficult for our models? Or was something else going on?

Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set.

_With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%. Scores measure Relative Human Action Efficiency (_ _RHAE_ ⁠(opens in a new window) _)_— _a metric comparing model performance to a human baseline. Based on_ _official gameplay logs_ ⁠(opens in a new window) _, we estimate the average human tester scored 48%. Models are not told how they will be scored, and cannot see their score throughout_— _actions only return a text representation of each frame and what level they are on._

ARC-AGI-3

ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions. You can play 25 demo games at arcprize.org/tasks⁠(opens in a new window).

ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks.

In gaming, GPT‑5.6 Sol has beaten Pokémon FireRed with a vision-only harness (as streamed by GPT\_Plays\_Pokemon⁠(opens in a new window)), Slay the Spire with Codex computer use (as streamed by EpochAI⁠(opens in a new window)), and the first stages of Baba Is You (as shared by Piotr Migdał & Piotr Grabowski⁠(opens in a new window)). What was so different about ARC-AGI-3?

!GPT-5.6 Sol in Codex completing a randomized daily challenge in Slay the Spire 2.

_Ethan Mollick_ _shows_ ⁠(opens in a new window) _GPT‑5.6 Sol in Codex beating a randomized daily challenge in Slay the Spire 2, a game released after GPT‑5.6 Sol’s knowledge cutoff._

Inspired by ARC’s analysis of GPT‑5.5’s shortcomings⁠(opens in a new window), we examined some of the GPT‑5.6 Sol’s attempts. Like ARC, we saw that the model didn’t appear too bright. It dwelled a long time on each action and struggled to make progress.

But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness.

First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.

Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT‑5.6 Sol was struggling to learn over time.

Agents do best when they remember what they’ve done

Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue.

!Diagram showing how model reasoning, tool calls, conversation history, and context compaction work together across a long-running task.

This is how our models are trained, and also how they are deployed in ChatGPT and Codex. To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API⁠(opens in a new window). Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.

With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent...