이미지: The Decoder
Summary
- Artificial Analysis has launched the "Search Index," which evaluates seven search APIs for AI agents on quality, cost, and speed
- Testing the same GPT-5.6 Luna model while swapping only the search provider, Parallel, Exa, and Firecrawl ranked at the top with scores of 75, 74, and 73 respectively
- Higher search quality was found to reduce the number of tokens agents use, lowering overall task cost as well
- 벤치마크명
- Search Index
- 개발사
- Artificial Analysis
- 테스트 모델
- GPT-5.6 Luna (모든 제공자 공통 적용)
- 테스트 프레임워크
- Stirrup (아티피셜 애널리시스의 오픈소스 에이전트 프레임워크)
- 테스트 대상
- Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, Brave (7개 제공자)
- 점수 범위
- 검색 미사용 33점 → 검색 사용 65~75점
- 상위 3사 점수
- Parallel 75, Exa 74, Firecrawl 73
- 비용 비교
- Parallel Search(advanced) 과제당 총비용 $0.084, Basic $0.11
Agents score 33 without search, up to 75 with it
How far do AI agent scores drop when search is removed? According to Artificial Analysis's new benchmark, the Search Index, the answer is 33. Attach a search API to the same model, and the score jumps to between 65 and 75. Search alone accounts for nearly half of an agent's performance.
Artificial Analysis doesn't build models itself. Instead, it's a scoring company that runs models through a battery of tests, combines the results into a single score, and measures speed and price alongside it. The "index score" figures that get cited every time a new model launches usually come from this company. On August 16, it also released Optima, which lets users build custom benchmarks with their own data. The new Search Index narrows that scoring business down to one specific domain: search APIs.
Why score search APIs separately
AI agents — systems that don't just answer questions but also search, generate files, and even make reservations — rely on external search APIs to find up-to-date information or facts scattered across the web. The problem is that result quality, response speed, and pricing structures vary widely across search providers. Depending on which search API a developer plugs in, the same model can see dramatically different task success rates and costs. Until now, there was no standard test to compare these side by side.
The Search Index targets that gap. It compares seven providers — Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave — by swapping only the search provider while keeping the same GPT-5.6 Luna model. Agents run on Artificial Analysis's open-source framework, Stirrup, performing 25 rounds of search and webpage retrieval per task.
Three tests combined into one score
The Search Index score is the sum of three equally weighted benchmarks. DeepSearchQA consists of 900 research-style questions that require multiple search queries to answer. A BrowseComp subset covers 200 difficult facts that can only be found through multi-step browsing. AA-Omniscience includes 600 questions spanning six knowledge domains. A "no-tool" baseline, in which the model answers without search, is added on top to measure how much a search API actually helps.
| Item | Score |
|---|---|
| No search (baseline) | 33 |
| Firecrawl | 73 |
| Exa | 74 |
| Parallel | 75 |
Better quality also means lower cost
Notably, better search results reduce the number of tokens a model uses to find an answer. The advanced tier of Parallel Search cut token usage by more than 40% compared to its Basic tier. While the per-query search cost is higher for the advanced tier, the total cost of completing an entire task was actually lower — $0.084 for advanced versus $0.11 for Basic.
A similar reversal shows up in speed. Parallel Search's turbo tier responds to a single query in 0.51 seconds, twice as fast as Basic's 1.03 seconds. But its quality score of 67 trails Basic's 73, meaning the agent has to repeat searches more often to get the answer it wants. As a result, the total time to complete a task ends up similar between the two tiers. Artificial Analysis named Parallel, Firecrawl, and Parallel (turbo) as the combinations with the best balance of cost and performance.
Participation remains open
The Search Index launched with seven providers, but it isn't a closed benchmark. Other search API providers can apply to Artificial Analysis to join the index, and the full scoring methodology is available to anyone in the published methodology document.
Editor's take
What makes this benchmark notable is that it scores a "component," not a model. Until now, AI performance comparisons have almost always been model-versus-model — is GPT better than Claude, how much smarter is this version than the last. But for developers actually running agents, the choice of peripheral components — search APIs, code execution environments, document parsers — matters just as much as model selection. Given that Artificial Analysis released Optima last week for building custom benchmarks with your own data, and followed it immediately with a search API scorecard this week, the company appears to be steadily breaking down what it scores from models into individual agent components.
Anyone who has worked with search APIs in practice will find these results familiar. Choosing a fast-responding search engine only to have its result quality fall short, forcing the model to re-run the same query three or four times, is a common experience that ends up increasing both total time and cost. The finding that Parallel Search's turbo and Basic tiers converge on similar total task time captures exactly that pattern in numbers. It means you shouldn't choose an API based on per-second response speed alone.
Teams in Korea building search-based agents would do well to measure the total per-task cost of the search API they currently use. The real cost only becomes visible when you look at total token usage and retry counts across a full task, not just the cost of an individual query. This index is likely to add more participating providers in the coming weeks — open benchmarks tend to attract applications from companies eager to rank at the top — and if that happens, the Search Index could become the default reference table for the search API market.
