Summary
- The Allen Institute for AI (Ai2) has released BenchMIRT, a methodology for auditing LLM benchmarks question by question.
- Analyzing 100 models, 16 benchmarks, and roughly 34,000 questions, the method independently identified two separate axes: safety and general reasoning.
- It found that BBQ, which measures social bias, and WMDP, which measures hazardous knowledge, correlate more strongly with reasoning ability than with safety.
Is the benchmark really measuring safety
The Allen Institute for AI (Ai2) released BenchMIRT on September 1, a new methodology for auditing LLM (large language model) safety and performance benchmarks at the question level. The data and per-model statistics used in the analysis were posted together in a BenchMIRT collection on Hugging Face, the model-sharing marketplace, and the code to run it was also published on GitHub.
To put it plainly, the UK AI Safety Institute made a similar case back in August, using psychometric techniques to analyze eight safety benchmarks and finding that they conflated several distinct traits — refusal strictness, honesty, and context-dependent handling — into a single score. Ai2's BenchMIRT takes that same concern further, independently identifying two separate axes, safety and general reasoning, across all 16 benchmarks and roughly 34,000 questions.
The audit starts from an old question: what does a single benchmark score actually measure? BBQ, for instance, was built to catch social bias, but one of its actual questions describes a grandfather and grandson booking an Uber and asks about age bias. Answering it correctly takes more than avoiding age stereotypes — it also requires tracking exactly who is who and reasoning through the evidence. A question that was supposed to test safety alone turns out to be testing reasoning ability at the same time.
An audit method borrowed from psychometrics
BenchMIRT extends item response theory (IRT), a tool from psychometrics — the field that estimates a person's abilities or traits from patterns in test responses. IRT starts from the premise that not every question gives the same amount of information about a test-taker: some questions are harder, and some are better at distinguishing skill differences than others. Ai2 had previously applied single-dimensional IRT to individual benchmarks in its "fluid benchmarking" research; BenchMIRT extends that into multidimensional IRT (MIRT), which can tease apart multiple abilities even when they're bundled into a single question.
The scale of the analysis is substantial. The team trained on results from 100 LLMs across 16 benchmarks and roughly 34,000 questions. Six of those benchmarks — MMLU-Pro, BBH, GPQA, IFEval, MATH, and MuSR — measure general reasoning, while the other ten, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest, were drawn from Ai2's Olmo 3 safety evaluation suite. Without being told in advance what any benchmark was supposed to measure, BenchMIRT found the safety and general-reasoning axes on its own, and the researchers said the same two axes kept turning up even when the analysis was rerun from scratch.
Most benchmarks measured what they claimed — a few didn't
| Benchmark | Correlation with reasoning | Correlation with safety | Audit finding |
|---|---|---|---|
| MMLU-Pro | 0.97 | -0.21 | Measures reasoning as intended |
| BBH | 0.94 | -0.20 | Measures reasoning as intended |
| GPQA | 0.81 | -0.12 | Measures reasoning as intended |
| IFEval | 0.72 | -0.34 | Measures reasoning as intended |
| MATH | 0.70 | -0.41 | Measures reasoning as intended |
| WildJailbreak | 0.14 | -0.90 | Measures safety as intended (250 benign questions lean toward reasoning) |
| HarmBench | 0.32 | -0.90 | Measures safety as intended (excluding copyright questions) |
| StrongReject | -0.16 | -0.84 | Measures safety as intended |
As the table shows, the six benchmarks built purely for reasoning, along with most safety benchmarks like WildJailbreak and StrongReject, correlated strongly with the ability they were designed to measure. The problem was the benchmarks flagged as exceptions.
BBQ and WMDP turned out to measure reasoning, not safety
BBQ is usually classified as a safety benchmark that measures social bias, but the BenchMIRT analysis found it links far more strongly to general reasoning than to safety. In other words, a model that scores low on BBQ may not actually be biased — it may simply be worse at understanding and reasoning through the questions.
WMDP is an even stranger case. It's a benchmark that tests how much dangerous dual-use knowledge a model holds in areas like biology, chemistry, and cybersecurity — knowledge that could be misused to create biological hazards or breach systems. BenchMIRT found that WMDP scores, too, correlate more strongly with general reasoning than with safety, but in the opposite direction: the stronger a model's reasoning, the lower its WMDP score tends to be. That's because the benchmark is designed to score refusing or failing to answer hazardous questions as the "desirable" response.
Even within HarmBench, the signals diverge
HarmBench shows that mixed signals can hide inside a single benchmark. Basic questions asking a model to write a phishing email to steal bank information, and contextual questions that set up a scenario to coax the model into misusing that information, both correlated strongly with safety. Copyright-related questions, though — like asking a model to reproduce Louis Armstrong's song lyrics verbatim — correlated much more weakly with safety and leaned closer to reasoning instead. Ai2 said this doesn't mean the benchmark itself is flawed; rather, multiple signals are bundled into a single score, and BenchMIRT untangles them to make the results easier to interpret.
The data and code are publicly available
The roughly 34,000 questions used in the evaluation, along with per-model and per-question statistics and the code, are all publicly accessible. The data and model statistics are available in the huggingface.co/collections/allenai/benchmirt collection, and the code to run it is published in the github.com/allenai/BenchMIRT repository. The collection is split into separate datasets: an evaluation set of roughly 34,300 questions, statistics for the 100 models, and statistics for roughly 29,600 questions.
Editor's take
What stands out in this release is how often we're seeing confirmation that scores labeled "safety" are actually measuring different things entirely. Back in August, the UK AI Safety Institute used psychometric techniques to analyze eight safety benchmarks and found three tangled signals — refusal strictness, honesty, and context-dependent handling. Now Ai2, in the US, has arrived at the same conclusion using a different methodology. Two independent research teams reaching the same finding through different methods suggests this isn't one team's optical illusion — it's a structural limitation baked into how these benchmarks are designed.
From a practical standpoint, this isn't just academic curiosity. Teams evaluating or procuring models often set a threshold like "safety benchmark score above X," but assuming a model is biased just because its BBQ score is low can be flat-out wrong — the model may simply lack the comprehension needed to work through the questions. Conversely, a highly capable reasoning model that scores low on WMDP might actually be a sign that it's refusing more safely, not less. Any team making procurement or deployment decisions based on a single overall benchmark score would do well to use this result as a reason to dig into which sub-questions actually drove that score.
In the months ahead, other labs and big tech companies will likely apply similar question-level audits to their own safety benchmarks and publish the results. Open-model developers, in particular, often lean on benchmark scores for marketing, so if audit tools like BenchMIRT become a standard part of validation, we can expect a wave of reinterpretation across many of the safety scores published so far.
Comments