이미지: The Decoder
Summary
- Artificial Analysis has launched Optima, a platform that lets users build custom benchmarks from their own data and workflows
- Data can be gathered three ways: uploading existing evaluation datasets, connecting agent trace tools, or installing a skill during coding sessions
- Early testers used it to build a benchmark for finance and accounting agents and found a model that cut costs by a factor of ten with no quality loss
- 제품명
- Optima
- 개발사
- Artificial Analysis
- 핵심 기능
- 사용자 데이터 기반 맞춤 벤치마크 생성 및 모델 비교
- 비교 축
- 품질, 과제당 비용(cost per task), 과제당 시간(time per task)
- 데이터 입력 경로
- 자체 파일·허깅페이스 데이터셋 / Arize·Braintrust·Langfuse 에이전트 트레이스 / 코딩 세션 스킬
- 채점 방식
- 루브릭 기반 평가, 페어와이즈(쌍대 비교) 평가 2종
- 공개 사례
- 재무·회계 에이전트 벤치마크로 비용 10분의 1 절감 모델 발견
Why a single benchmark score wasn't enough
When choosing an AI model, people commonly rely on a single composite score, like an IQ-style rating. But once deployed on real work, the highest-scoring model isn't always the right answer. If a model needs several retries to complete the same task, or its output requires human correction, the actual cost can end up higher even if the per-token price is cheap. Artificial Analysis has released Optima, a platform aimed squarely at this problem.
Artificial Analysis is an independent evaluation company known for benchmarks like GDPval-AA and AA-Briefcase. Optima isn't a general-purpose benchmark; it's a tool that pits models against each other using the user's own data and workflows. According to the company, users can build a benchmark from their own data, workflows, or a description of a desired scenario, then run it against today's leading models to compare quality, cost per task, and time per task side by side.
Three ways to start, even without existing data
Optima offers three paths for gathering material. Users who already have an evaluation dataset can import it directly from their own files or from Hugging Face. Teams running agents can pull records from trace tools like Arize, Braintrust, or Langfuse. Developers can also install a skill in their coding environment that automatically collects data from past sessions.
| Input path | Target users | Method |
|---|---|---|
| Existing evaluation dataset | Teams that already have data | Upload own files or from Hugging Face |
| Agent traces | Teams running agents | Connect via Arize, Braintrust, or Langfuse |
| Coding session skill | Developers | Install skill in coding environment, auto-collect past sessions |
Having no data at all is fine too. Users can describe the scenario they want to test and provide a few example inputs and outputs, and Optima will generate suggested test inputs, evaluation criteria, and sample tasks. Users then review these, refine them with feedback, and run the actual benchmark.
Two scoring methods, three result metrics
There are two scoring approaches. One is a rubric-based method that grades answers against objective criteria. The other is the pairwise comparison method that Artificial Analysis also uses in GDPval-AA and AA-Briefcase. In the pairwise approach, users first look at a subset of response pairs and mark which one is better; Optima then uses that preference data to rank the entire dataset.
The comparison results don't stop at a single quality score. Cost per task and time per task are presented as independent comparison axes alongside it. In agentic applications, per-token pricing alone doesn't reveal the real cost. Even a cheap model can end up costing more overall if it requires more attempts, fails often, or needs human correction of its output. The actual cost incurred per completed task is the more honest number.
A real-world use case
According to Artificial Analysis, some early testers built a benchmark for finance and accounting agents and found a model that could cut costs by a factor of ten without significantly degrading quality. This can be seen as a working example of the exact problem Optima was designed to address: finding the optimal model for a specific work environment that general-purpose benchmarks fail to reveal.
How to try it
Optima is now available. Following the process the company described, users can build a benchmark through the steps below.
- Choose your material — Import an existing evaluation dataset from files or Hugging Face, or connect Arize, Braintrust, or Langfuse traces if you run agents. Coding teams can install a skill in their coding environment to automatically gather data from past sessions.
- No data? Start with a description — Write out the usage scenario you want and provide a few example inputs and outputs. Optima will suggest test inputs, evaluation criteria, and sample tasks.
- Refine the suggestions — Review the proposed test setup and adjust it with feedback until it fits your needs.
- Choose a scoring method — Pick between the rubric method, which grades against objective criteria, and the pairwise method, which ranks based on compared response pairs.
- Run and compare — Run the benchmark against today's leading models and view quality, cost per task, and time per task side by side.
Specific usage conditions such as country availability, pricing plans, or platform restrictions were not specified in this announcement. Still, a few applications come to mind within the scope described. A team running a customer support agent, for instance, could feed in actual support logs as traces to see which model maintains response quality while cutting processing time. A coding team could feed in past session data gathered via the skill to compare which model has a lower failure rate on tasks like refactoring or bug fixes.
On August 10, Artificial Analysis posted on its official X account that it was preparing a new lineup of AI benchmarking products for developers and was accepting early access applications. This Optima launch appears to be the realization of that lineup.
Editor's take
This launch captures a broader shift in how AI models are chosen: from "what's its IQ score" to "how cheaply and quickly does it get my work done." Just days after news that Grok 4.6 had matched GPT-5.6 Sol with an IQ score of 61, the very company behind that ranking is now essentially admitting that the score alone isn't enough. The fact that Grok 4.6 matched or beat nearly every model except Claude Opus 5 on practical benchmarks while costing far less per task already showed that composite scores and real-world cost efficiency don't move in lockstep. Optima is essentially an invitation to measure that gap yourself, at the individual or company level.
In practice, the same pattern keeps repeating when models are put to real work. The top-ranked model on a leaderboard often turns out to need repeated corrections or human intervention to fix formatting on certain tasks. Conversely, a lower-ranked model frequently nails a specific domain task in one pass without errors. A composite score is just an average — there's no guarantee your particular workload falls anywhere near that average.
Rather than rolling this tool out company-wide immediately, domestic companies would do better to pilot it on a single workflow first. Picking one task where failure has a visible cost — say, customer response or document summarization — and measuring cost and time per task can show, in hard numbers, whether the model currently in use is really the best choice. That said, a poorly designed benchmark still produces poor results. Since the entire ranking in pairwise evaluation hinges on how the first few response pairs are judged, it's safer for teams to agree on their initial judging criteria in advance.
In the coming weeks, other evaluation firms and cloud providers are likely to roll out similar "custom benchmark" tools of their own. The benchmarking market itself is shifting — from "who can build the more accurate test" to "who can write the test that fits my specific job for me."
