Tutorial: Benchmarking GPT-6 Astra vs Claude Fable 5.1 vs GPT-5.6 Sol using W&B Weave

Benchmarking GPT-6 Astra, Claude Fable 5.1, and GPT-5.6

Announcement

Webinar

Podcast

GTC 2026

Three-time Platinum. Still the only one. CoreWeave earns SemiAnalysis’ highest ClusterMAX™ rating for the third consecutive time. Read more ‍

Products

Data and storage

Infrastructure control

Runtime acceleration

Model and agent development

Mission control

Solutions

Pricing

Resources

About us

Contact us Login

Contact us Login

Clear

Originally published on the Weights & Biases by CoreWeave blog on September 7, 2026. GPT-6 Astra is here. OpenAI’s latest model brings improvements in coding, research, and tasks that require multiple steps to complete. With most new model releases, benchmark scores tend to be relatively close. This release is different, with Astra posting benchmark results that are well ahead of the competition. Looking at the public benchmarks alongside Claude Fable 5.1, which also focuses on coding and token efficiency, gives us a useful comparison. We also bring in GPT-5.6 Sol, the predecessor Astra is meant to replace, since a fair read on progress needs the model being succeeded, not only the closest rival. Beyond benchmark scores, there are two important questions to ask: how many tokens does each model use to solve the same problem, and how much does it cost? In this tutorial, we will look into GPT-6 Astra and its benchmark results, then use Weave to compare it with Claude Fable 5.1 across 50 HumanEval coding tasks. We will test their generated code and track token usage, response time, and estimated API cost. Introduction to GPT-6 Astra OpenAI introduced GPT-6 Astra on September 3, 2026. As with every major model release, OpenAI positions it as its most capable model yet. As of today, GPT-6 Astra stands out as one of the strongest models for Agentic tasks , with the ability to take complex work from an initial instruction through to a finished result. This includes software engineering, research, computer use, and creating documents, spreadsheets, and presentations. Most of the developers have produced good results in generating static games and 3 animated apps, rendering decent results. For developers, GPT-6 Astra is available through the API under the model ID gpt-6-astra . It supports a 1.05 million token context window and can generate up to 128,000 output tokens. The release gives us an early look at Astra’s performance. On Terminal Bench 4.0 , which evaluates Agents on terminal based tasks such as software engineering, system configuration, and data analysis, OpenAI GPT-6 Astra reports a score of 57.9%. Claude Fable 5.1 scores 55.8%. OpenAI reports that Astra costs 63% less per task than Claude Fable 5.1 in this evaluation. Since these figures come from OpenAI’s own testing, the results are best understood within the specific setup and conditions of the benchmark. It also leads on DeepSWE and Terminal-Bench Science. Fable 5.1 scores higher on Humanity’s Last Exam with tools, reaching 65.0% against Astra’s 57.2%. It also leads the Artificial Analysis Intelligence Index with 65.7 against 61.2. The results do not show one model winning every type of work. Two Astra results stand out because they are close to the limits of their benchmarks. Astra (Max) scores 99.9% on ARC-AGI-3 , which tests how well a model learns and acts in unfamiliar interactive environments. It also records 0% on the ExploitGym honeypot evaluation, where a lower score is better. Results this close to a benchmark boundary raise an important question: can the test still show meaningful differences between future models? We will return to that question after reviewing the API prices. Here are the standard API prices for Claude Fable 5.1 , GPT-6 Astra, and GPT-5.6 Sol as of September 7, 2026:

Figure 1: Model Pricing - OpenAI and Anthropic Astra and Fable both charge $10 per million ordinary input tokens and $50 per million output tokens, while Sol sits at half that rate on every column. For a short coding task, the main cost difference should come from the number of tokens each model uses, since the per token price of Sol is already fixed lower. Are benchmarks saturated?

Figure 2: Public LLM benchmark saturation The latest model releases are pushing several established benchmarks close to their limits. At that point, the score still shows that a model can perform the task. However, it provides less information about the difference between leading models. A result of 99.9% leaves almost no room for the next model to show progress. It may become faster, cheaper, or more reliable on real work without producing a visible gain on that benchmark. The test served its purpose, but leading models now need a harder test. The score can also hide the amount of compute used to reach it. A model may solve more problems when it receives a larger reasoning budget, more attempts, or a longer time limit. There is one important subreddit discussion on FrontierMath Erdős , for example, limits spending to $300 per problem. Astra can solve more of those problems when it runs beyond that budget, but the result then answers a different question. One score measures performance within a cost limit, while another measures the model’s maximum capability. On the same line, the researchers at ICML 2026 put a number on this. In their paper, When AI Benchmarks Plateau they analyzed 60 benchmarks drawn from the technical reports of major model developers. Nearly half of them, 29, had saturated, and in 14 of those, the scores at the top were so close together that the leading models could no longer be ranked against each other. The ARC Prize team published two scores for Astra rather than one. Under its Standard harness, which gives every model the same minimal interface, Astra scores 62.7%. Under the Provider Adapter harness, which lets Astra keep its own reasoning state between requests, the same model reaches 99.9%. Both scores are verified and both are state of the art. The second leaves a tenth of a point of room, so the next model cannot show progress there even if it does better work. ARC Prize will now publish both conditions with the harness labelled, which is the practical answer to saturation: once a score approaches the ceiling, the setup that produced it carries more information than the number itself. The team is already working on what the next generation should measure, including recursive self-improvement and open-ended innovation. Does a higher benchmark score mean a lower cost per task? A...