Astra's 99.9% Score Came From the Harness, Not the Model

ARC-AGI-3 works differently from its two predecessors. Instead of testing whether a model can produce a single correct answer, it drops the model into a game whose rules it doesn't know and measures how quickly it figures those rules out and clears the level. That means the score hinges on how much of what the model learned on one move it can carry forward to the next.

A harness is the outside program that manages that carry-forward. ARC Prize's standard harness is built to attach identically to any company's model, so it only passes along whatever notes the model chose to write down for itself. OpenAI's provider-adapter harness, submitted alongside its results, works differently: it preserves the model's entire internal reasoning state across requests, compressing and splicing it together as the conversation runs long.

Think of it as a test room. With the standard harness, the proctor collects your answer sheet after every question and lets you carry only a single handwritten note into the next one. With the provider adapter, you get to carry your entire working memory — the calculations you built up solving the last question — straight into the next. It's no surprise the same test-taker scores differently under those two setups.

The effect wasn't limited to the score. On the 167 games both harnesses solved, ARC Prize measured the adapter running 3.66 times faster while using 49% fewer tokens. Cost flipped too: at the highest reasoning setting, the standard harness cost $26,098 versus $17,332 for the adapter.

The cost figures are worth flagging on their own. ARC Prize also converted its human baseline into dollar terms, and once you factor in time compensation, a human clears one game for $12.78. The $26,098 and $17,332 Astra ran up at its highest setting cover the whole test set, so a direct comparison isn't fair — but the fact remains that, for now, a human is still the cheapest way to pass this test.

Opus 5's 30.2% Tells the Same Story

Claude Opus 5's 30.16% score, listed right next to Astra's, is getting cited everywhere as a point of comparison. When Opus 5 was submitted on July 24, it was the top ARC-AGI-3 score, nearly four times the 7.8% that GPT-5.6 Sol had set before it.

But there are already two other records of the same Opus 5 wrapped in a different shell. According to The New Stack, NVIDIA wrapped Opus 5 in its own AVO agent system in August and hit a human-relative action-efficiency score of 100.00 across 183 levels. AWS did the same with its Strands agent, also across 183 levels, and scored 99.95 — though a single run took eight hours and $830 in token spend, during which the agent wrote 734 Python scripts on its own as parsers and simulators for individual games. In 10 of 25 games, it simply reused scripts it had already written.

Bare, 30%. Wrapped, 100%. Opus 5 already demonstrated that gap last month; Astra just showed the same gap in a single results sheet this time. The AWS team put it plainly: benchmark scores were measuring the absence of a harness, not the limits of the model.

So What Did Astra Actually Do Well

Even after setting the wrapper story aside, something genuine remains. ARC Prize recorded that Astra solved games by writing out game states in its own shorthand notation — not notation pulled from its training data, but shorthand it invented fresh for each game. ARC Prize described it as emergent shorthand rather than a programming language.

Its action efficiency also stands out. Astra cleared 96.0% of all levels in fewer moves than the human baseline, averaging 51.7% fewer actions per level. That human baseline comes from roughly 500 general participants playing the same games. On the earlier generations of the test, Astra scored 97.5% on ARC-AGI-1 and 95.0% on ARC-AGI-2.

ARC Prize also drew a clear line: saturating this test isn't evidence that AGI has been achieved, and ARC-AGI-3's rules are deterministic and closed, so they can't stand in for the messiness of the real world.

The Standard-Harness 62.71% Is a Record in Its Own Right

It's easy to lose this under all the adapter debate, but Astra's 62.71% score on the standard harness — the one where everyone competes under the same conditions — is itself a record. It's more than double Opus 5's 30.16% from July 24, and eight times GPT-5.6 Sol's earlier 7.8%. Unlike previous generations, the score also doesn't collapse sharply when reasoning effort is dialed down: below the top setting's 62.71%, XHigh scores 59.34% and High scores 54.82%.

General benchmarks tell a far smaller story. Astra's Artificial Analysis Intelligence Index score of 61.2 only barely edges out GPT-5.6 Sol's 60.9. Pricing sits at $10 per million input tokens and $50 per million output tokens, in line with Claude Fable.

That combination — flat general scores paired with a huge jump on an interactive test — is exactly why interpretations of this result are split. Epoch AI's Greg Burnham called it the end of one era and the start of another.

Editor's Take

The most valuable line in this results sheet isn't the 99.95% — it's the fact that it sits right next to 62.71%. ARC Prize neither hid OpenAI's adapter result nor let it stand in for the standard one; it published both. That's what lets us, for the first time, separate the model's power from the wrapper's power in the same table.

Anyone who's followed benchmarks for a while will find this pattern familiar. It's the same dynamic as a coding model's score jumping once you bolt on an agent loop, or a math score jumping once you sample multiple answers and take a majority vote. What's different this time is that the model's own maker built and submitted the wrapper, and no other company's model got to use it. The New Stack pointed out that the models being compared were measured under different setups, and that criticism holds up.

The lesson for teams building on these models is pretty practical: reconsider how you split your time between picking a model and building the system that wraps it. In the AWS case, what generated the score wasn't a bigger model — it was infrastructure that exported logs to files, pulled its own previously written code back out, and stepped in when something got stuck. At $830 and eight hours, that's a scale most teams could reproduce in-house.

It's also time to change how we read benchmark tables. Going forward, every score needs to come with a note on which harness produced it, and any number without that note isn't usable for comparison. Astra is genuinely a strong model. And this 99.95% proved something beyond that strength: how much of the outcome depends on the machinery that lets that strength be used to its full extent.