이미지: @AIatMeta (X) 화면 갈무리
Summary
- Meta AI introduced its internal evaluation framework WildArtifactBench, releasing an initial set of 10 tasks.
- Rather than grading against a fixed answer key, outputs are compared using win rates and Elo scores judged by humans and other agents.
- The tasks range from analyzing bird calls to reading coronary artery stenosis, mixing in practical, real-world work.
- 명칭
- WildArtifactBench
- 발표
- 메타 AI 공식 X 계정, 2026-08-20 18:25 UTC
- 형태
- 내부 평가 프레임워크(프리뷰)
- 공개 과제 수
- 10개
- 평가 방식
- 사람·에이전트 심사자의 승률(win rate)과 Elo 점수
- 과제 구성
- instruction.md · data 폴더 · rubrics.json
- 예시 과제
- Bird Sound(오디오 분석), Coronary Stenosis(관상동맥 협착 판독) 등
How Do You Grade a Task With No Fixed Answer?
This time, Meta AI didn't unveil a new model — it unveiled a grading standard. In a post on its official X account on the 20th (local time), Meta AI introduced for the first time WildArtifactBench, an internal framework for evaluating the real-world capabilities of AI agents. Instead of the usual performance table that accompanies every new model release, this time the star of the show is the tool used to measure that performance.
Agents no longer just produce a single answer. They write documents, generate 3D models, edit video, and read medical images. The problem is that these outputs don't have a fixed correct answer. They can't be judged as simply "pass or fail" the way code can be run and checked, nor scored right or wrong by a single number the way math problems can. Meta AI described WildArtifactBench as designed to evaluate "complex, real-world-like tasks across a wide range of output formats."
Win Rates and Elo Scores Instead of Fixed Answers
The grading method is unusual. Instead of a strict ground-truth rubric, human judges and other AI-agent judges pit outputs against each other head-to-head, scoring them by win rate and Elo rating. It borrows the same Elo rating system used in chess and online game rankings to determine which agent's output is relatively better. Meta AI explained that this approach "expands task coverage across practical, multimodal workflows." The 10 tasks released so far were described by Meta AI as a step toward "further advancing the ability to measure the practical value that multimodal agents actually deliver."
The 10 Released Tasks
| Task Name | Category |
|---|---|
| Bird Sound | Audio analysis |
| Cardiac Ultrasound | Medical/healthcare |
| Cat Mesh | Digital design |
| Coronary Stenosis | Coronary angiography image analysis |
| Dqn Driving | Machine learning engineering |
| Kitchen Layout | Digital design |
| Materials Metrology | Quantitative image analysis in materials engineering |
| Microchip Model | 3D modeling |
| Plywood Whale | Digital design |
| Video Editing | Video editing |
Each Task Splits Into Three Parts
According to screenshots from the released task viewer, each task is made up of three components: an instruction.md file spelling out what to build, a data folder containing input materials such as images, YAML files, or audio, and a rubrics.json file laying out the grading criteria. The viewer also shows four tabs — Category, Input files, Input size, and Criteria — letting users compare input formats and difficulty across tasks.
How to Try It
WildArtifactBench can be accessed through the task viewer on Meta AI's developer page. The list of 10 tasks appears on the left side of the screen, with a search bar at the top for finding tasks by name.
- In the task viewer, select a task you want to examine (e.g., Bird Sound, Coronary Stenosis).
- Once selected, the screen displays the Category, Input files, Input size, and Criteria tabs.
- Under the Task files tab, you can open instruction.md, the data folder, and rubrics.json individually. Images load automatically, YAML files must be expanded to view, and audio files only display metadata.
- Clicking the full task archive link at the top of the screen lets you download all 10 tasks at once.
The announcement did not include details on external developer access scope or participation requirements. However, given that Meta has recently rolled out a string of long-horizon, task-oriented agents such as the terminal coding agent Muse Code, tasks like Dqn Driving or Microchip Model could end up serving as a testing ground for measuring the real-world performance of such agents.
Editor's Take
Companies like OpenAI and Anthropic also build their own benchmarks, but most stay confined to domains with clear correct answers, like coding and math. Meta expanding its evaluation scope into domains without fixed answers — medical image reading, 3D modeling, video editing — reads as a signal that the center of gravity in agent competition is shifting from "who solves the problem better" to "who produces output closer to real-world use." Bringing in an Elo-style approach for domains where ground-truth grading is impossible fits the same trend. Going forward, the center of gravity for benchmarks may well tilt from measuring "who is more accurate" toward measuring "who is more preferred."
Applying existing agent benchmarks to real-world work always runs into the same wall. Coding benchmarks measure "does it pass the tests," but not "would a team actually adopt this code." A win rate/Elo approach is, at minimum, a first step toward closing that gap.
For teams in Korea building their own agents or considering adoption, what matters more than a single score on a performance table is understanding how that score was derived. An agent that excels at tasks with fixed answers and one that excels at producing real-world outputs without fixed answers require different capabilities. Building a small, separate evaluation set tailored to your own company's document formats and code style will be more useful for actual adoption decisions than an external benchmark score.
If Meta expands the scope of WildArtifactBench beyond its initial 10 tasks, the next likely step is using this framework to directly publish performance results for its own agents, such as Muse Code.
Comments