이미지: X — 벤치마크·평가 화면 갈무리
Summary
- Artificial Analysis has released AA-AnalystAgent, a benchmark for quantitative-analysis agents that work with real spreadsheets and documents; Claude Opus 5 topped the leaderboard at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49%.
- On pass@1 (getting it right in a single attempt), GPT-5.5 (xhigh) led with 66%, but on pass^5 (succeeding all five times), Claude Opus 5 came out ahead.
- An analysis of 1,567 failed attempts found that 57% involved "clinging to an initial, incorrect hypothesis," and models scoring the same 20% differed in per-task cost by as much as $0.05 versus $1.34.
- 벤치마크명
- AA-AnalystAgent (Artificial Analysis 개발)
- 평가 대상
- 실제 업무용 스프레드시트·문서 기반 정량 분석 에이전트 과제
- 상위 점수
- Claude Opus 5 54%, GPT-5.5 50%, Claude Fable 5 49%
- pass@1 최고
- GPT-5.5(xhigh) 66%, Gemini 3.1 Pro Preview·Claude Opus 5(max) 각 64%
- 실패 분석
- 10개 모델의 실패 시도 1,567건을 7개 실패 유형으로 분류, 초기 가설 집착이 57%
- 비용 격차
- Claude Sonnet 4.6(max)와 MiMo-V2.5-Pro 모두 20%, 작업당 각 1.34달러·0.05달러
- 최고 비용 모델
- Claude Opus 4.7(max), 작업당 1.98달러
- 오픈 가중치 1위
- Kimi K3(max) 39%, 비공개 프런티어 대비 15점 차
The same analytical task was run five times. Get four right and miss one, and the task scores zero. That's the grading method behind AA-AnalystAgent, a new benchmark from independent evaluator Artificial Analysis. Even at the top of the leaderboard, scores barely cleared the halfway mark. Claude Opus 5 finished first at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49%.
Not a workbook of questions, but a test with real files
Language model evaluation has so far mostly consisted of static text-based question-and-answer tests. AA-AnalystAgent is different. It hands models real-world spreadsheets and documents, requiring them to use tools to open files, extract figures, and complete the calculations themselves. This kind of tool-using test is known as an agent benchmark.
Artificial Analysis explained the rationale behind creating this task by pointing to the nature of analyst work itself. "In real analyst work, professional judgment and domain expertise matter just as much" as raw accuracy, the team noted. In other words, getting a single correct answer is a different skill from producing an analysis good enough to hand off to someone else.
| Model | Score | |
|---|---|---|
| Claude Opus 5 | 54% | |
| GPT-5.5 | 50% | |
| Claude Fable 5 | 49% | |
| Kimi K3 (max, top open-weight model) | 39% |
What decided the ranking wasn't raw performance but consistency
An interesting wrinkle is that single-attempt accuracy and final ranking don't line up. On pass@1, GPT-5.5 (xhigh) posted the highest score at 66%, with Gemini 3.1 Pro Preview and Claude Opus 5 (max) close behind at 64%. But on pass^5 — what the evaluation team calls "pass-all-5," requiring success across all five attempts — Opus 5 came out on top. The model that rose to the top wasn't the strongest performer, but the most consistent one.
| Metric | Leader | Value |
|---|---|---|
| pass@1 (probability of getting it right in one try) | GPT-5.5 (xhigh) | 66% |
| pass@1 joint runner-up | Gemini 3.1 Pro Preview / Claude Opus 5 (max) | 64% each |
| pass^5 (success on all 5 attempts) | Claude Opus 5 | Leaderboard leader |
The announcement post doesn't specify exactly how the headline 54% score was calculated. But the fact that pass@1 and the final score diverge in itself reveals something about what this leaderboard is actually measuring.
57% of failures came from "not letting go of the first hypothesis"
The evaluation team collected 1,567 failed attempts from 10 leading models and sorted them into seven failure categories, allowing multiple categories per attempt. The most common failure type was persisting with an incorrect hypothesis formed early on, which appeared in 57% of all failures.
It's a familiar scenario for anyone who's worked in analysis. Misread the data, set a direction based on that misreading, and then run every subsequent calculation along that flawed path. Human junior analysts do the same thing. The difference is that a person eventually pauses and says "wait, this doesn't add up," while a model tends to push through to the end with plausible-sounding logic.
Same score, 27x difference in cost
Overlaying cost onto the results paints yet another picture. Claude Sonnet 4.6 (max) and MiMo-V2.5-Pro both scored 20%, but their per-task costs were $1.34 and $0.05, respectively — a 27x gap for the identical result. The same pattern holds near the top: the most expensive model tested, Claude Opus 4.7 (max), cost $1.98 per task.
Among open-weight models, Kimi K3 (max) topped the field at launch with 39%. That's 15 points behind the leading closed frontier model, but ahead of all eight other open-weight models tested alongside it. DeepSeek's DeepSeek V4 Flash placed next.
So what changes now
On August 10, Artificial Analysis opened early access to what it described as a suite of developer benchmarking products in the works. AA-AnalystAgent sits within that effort — an evaluation that transplants office work directly into a test format. For organizations looking to put AI into roles where spreadsheets are the job itself, such as finance, accounting, or research, there's now one more data point to consider.
The message is simple: today's agents can pull off a task convincingly once, but not reliably five times in a row. That means it's still too early for humans to accept their output without review. The agent harness used to run the tasks — Stirrup, the execution wrapper that lets models open files and run code — has been released as open source on GitHub, so anyone can reproduce the results under the same conditions. It's an environment where fewer people need to simply take an announcement's word for it.
