Benchmark

A benchmark is a standardized test that AI models all take under the same conditions. Just as students are compared by their exam scores, models are compared by having them solve the same set of problems. Every number cited in a "which model is smarter" article traces back to one of these tests.

The subjects vary widely. General knowledge (MMLU), PhD-level science (GPQA), competition math (AIME), real-world coding (SWE-bench) — those unfamiliar strings of letters attached to the bar charts in new model announcements are all test names.

Two things to watch for when reading these numbers. First, most published scores are "self-reported" — measured by the company under conditions favorable to itself, so they can differ from third-party verification. Second, being good at taking tests isn't the same as being good at doing the job. Disputes over "benchmark contamination," where leaked questions inflate scores without real improvement, come up often too. So it's more accurate to check which test was used and who measured it than to fixate on a few-point gap.

"New model claims to beat rivals across all major benchmarks" — the word "claims" shows up because of the self-reporting problem above. METAL's benchmark section has explanations for each individual test.

1. Search for "LMArena" (Chatbot Arena) and go to the site — it's a blind comparison platform that shows answers from two AI models side by side without revealing their names.
2. Ask any question you like and vote for the better answer. After voting, the models' identities are revealed.
3. Millions of these votes accumulated together form the "Arena ranking" — a good way to see for yourself that exam scores (benchmarks) and how people actually feel about a model (Arena) can diverge.