이미지: The Decoder
Summary
- Moonshot AI released PerceptionBench, which isolates and measures pure visual perception ability in multimodal models
- None of the 16 frontier models tested exceeded 60%, with GPT-5.6 Sol topping the list at 59.7%
- The paper's authors said many failures previously classified as "reasoning errors" were actually already off-track at the image-reading stage
- 벤치마크 명칭
- PerceptionBench
- 발표
- 문샷 AI(Kimi 개발팀)
- 공개 문항 수
- 3,000개 (내부 검증 문항 17,000여 개 중)
- 측정 하위 영역
- 10개(Visual Relation, Counting, Attributes 등)
- 1위 모델
- GPT-5.6 Sol, 정확도 59.7%
- 상위 5개 모델 격차
- 4%포인트 미만
- 최하위권 모델
- GLM-4.6V, 정확도 32.5%
- 참조 분석 대상
- 기존 오픈소스 벤치마크 42개
Not even one model cleared 60%
OpenAI's GPT-5.6 Sol, Moonshot AI's Kimi K3, Anthropic's Claude Fable 5, Google's Gemini 3.1 Pro — gather 16 of today's top-performing multimodal models and test them on pure "looking," and not a single one clears 60%. Even the top scorer, GPT-5.6 Sol, managed only 59.7%. These were tasks a human could handle easily: telling where a clock's hands point, or counting how many flowers are inside a red box.
Moonshot AI released a new benchmark, PerceptionBench, containing these results. The company develops the Chinese AI assistant Kimi and is known for building large models while also releasing their weights openly. The core of this benchmark is that it strips away "reasoning" and "knowledge" to measure pure visual perception ability alone. Every question is designed to be answerable just by looking at the image, requiring no background knowledge or logical reasoning.
Even across 42 benchmarks, coverage barely overlapped
The researchers said that after analyzing 42 existing open-source benchmarks, they found that the error types each benchmark catches barely overlap with one another. This means each benchmark only covers a fragment of visual perception, and none of them alone can capture the full picture. So instead of defining categories upfront, the researchers gathered actual cases where models made mistakes and traced backward to find the earliest point where perception went wrong, extracting 10 "skill areas" from this process: Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
The internally validated question set exceeded 17,000 items, but only 3,000 of them were released this time. The researchers said 60% of these were drawn from actual cases where models made mistakes, while the remaining 40% were newly created by modifying images.
Results: top models clustered within 4 percentage points
| Model | Accuracy | |
|---|---|---|
| GPT-5.6 Sol | 59.7% | 60 |
| Kimi K3 | 58.5% | 59 |
| Claude Fable 5 | 57.2% | 57 |
| Gemini 3.1 Pro | 56.2% | 56 |
| GPT-5.5 | 55.8% | 56 |
| Qwen3.5-397B-A17B | 47.5% | 48 |
| GLM-4.6V | 32.5% | 33 |
As the table shows, the top five models are clustered within 4 percentage points, from 59.7% down to 55.8%. This means that models from different generations and different companies remain at roughly the same level when it comes purely to "seeing" ability. In contrast, open-source models fell further behind, with Qwen3.5-397B-A17B at 47.5% and GLM-4.6V at 32.5%.
What looked like "reasoning errors" were actually perception failures
The researchers placed particular emphasis on one finding. When models failed complex problems, the failure was typically explained as "a failure of logical reasoning." But in reality, the mistake had often already happened at the very first step — reading the image incorrectly. The researchers said they confirmed this through four case studies that broke multi-step problems down into individual perception questions.
Placed alongside other benchmarks released in August, the pattern becomes clearer. Cases such as the recovery of Claude and GPT's hidden reasoning tokens, which raised suspicions of benchmark memorization, and Epoch AI's undisclosed game-puzzle benchmark released on August 6 — where Opus 5 scored 59% — all point to the same underlying question: whether current benchmarks are actually measuring real capability. What makes PerceptionBench notable is that it pinpoints the most basic layer of all — the fact that models' ability to simply "see" is still incomplete.
Editor's view
What makes this result interesting isn't the ranking, but the size of the gap. While frontier models are being loudly announced as having already surpassed or approached human performance on language, coding, and math benchmarks, the most basic capability of all — accurately seeing an image — still hasn't crossed the 60% threshold. Moreover, the fact that the top five models are clustered within 4 percentage points suggests this isn't a shortfall specific to any one company's technology, but closer to a structural limitation of multimodal training methods themselves.
Anyone who has worked hands-on with vision-language models will find this result familiar. When asked to read tables in documents, count objects in photos, or spot subtle differences between two images, these models often produce plausible-sounding explanations while getting the actual answer wrong. Such failures have typically been lumped together as "weak reasoning," but PerceptionBench's conclusion suggests otherwise — that the image itself was likely misread from the start. When the diagnosis changes, so does the solution: this may not be a problem that more elaborate prompting to induce better reasoning can fix.
For Korean companies planning to deploy vision-language models for tasks like document review, quality inspection, or inventory checks, a human verification step to double-check the model's initial perception output should be a mandatory part of the process at this stage. Automation confidence should be set lower particularly in the areas PerceptionBench flagged as weak points — counting, fine-grained recognition, and localization. In particular, inspection automation that requires counting numbers or distinguishing fine attributes is not yet ready to fully do without human oversight.
In the coming weeks, other labs are likely to release their own vision benchmarks or responses. If a model emerges that first clears the 60% threshold, the next round will reveal whether that represents genuine progress in perception ability or simply reflects training data reinforced to match this particular benchmark.
