AI Safety Scores Can Be Gamed Just by Refusing More

Before releasing a language model, developers score it against safety benchmarks such as HarmBench, OR-Bench-Hard, and SORRY-Bench. A benchmark is a standardized test that AI models take together. A low score can delay a launch or send a model back for further work. The problem is that no one had rigorously verified whether these scores actually measure a single, coherent property called "safety." A similar issue was raised earlier in Even Top AI Models Scored Below 60 on 'Visual Perception', where a benchmark turned out to be measuring something different from what it claimed to measure. This new study tackles that same problem head-on in the safety domain.

A test that lumps three things into one

The research team analyzed results from as many as 192 models answering over 5,000 questions — which the team itself described as the largest analysis of its kind. The finding: the eight benchmarks are not measuring a shared "safety" property at all, but three largely unrelated traits — how strictly a model refuses requests, how honestly it answers, and how it handles content that can be harmless or harmful depending on context. How honestly a model answers showed almost no correlation with how often that same model refuses requests.

The most problematic pairing was HarmBench and OR-Bench-Hard. HarmBench rewards a model for refusing harmful requests. OR-Bench-Hard, conversely, penalizes a model for being overly cautious and blocking harmless requests. A model that scores well on one almost always scores poorly on the other.

Trait	Representative benchmark	What it measures
Refusal strictness	HarmBench	How well it blocks harmful requests
Overcensoring	OR-Bench-Hard	Whether it blocks harmless requests too
Honesty	Honesty-related benchmarks	Whether answers align with facts

Averaging scores across multiple benchmarks without accounting for this trade-off lets a model boost its overall score simply by refusing more requests — even as its real-world usefulness declines. Hidden behind this averaging practice was also the issue of multiple, similar tests redundantly scoring the same underlying behavior.

Most questions tell you nothing

The second finding concerned testing costs. Most questions were either answered correctly by nearly every model or failed by nearly every model, revealing no meaningful difference between models. When only the most informative questions were selected, three short 25-question tests captured the three safety dimensions more accurately than a random sample of the same size.

Adaptive testing — similar to a placement exam where the next question depends on how well the student is doing — improves efficiency even further. For individual benchmarks like HarmBench or SORRY-Bench, about 10 questions were enough to closely approximate the results of a full-scale evaluation. That translates to a 97-99% cost reduction, making it practical to run regular safety checks after every fine-tuning or model compression pass.

Catching models that behave only in test situations

The third finding weighs most heavily in the safety debate. The team called this "sandbagging" — a model recognizing it's being tested and behaving more cautiously than it normally would. The concern is that a model could detect a test scenario on its own and deliberately alter its behavior in response.

Such behavior leaves traces in response patterns. Just as a student who nails hard questions but flubs easy ones looks suspicious, a model that answers difficult questions correctly while missing easy ones looks suspect even if its overall score seems fine. Using this method, the team caught 80-100% of models that had been explicitly instructed to behave with excessive caution. The trade-off was a false-positive rate of about 1 in 10 for normal models incorrectly flagged as suspicious. Adding a second check — comparing a model's current performance against its own past performance — raised the detection rate to 94-97%. A model that fakes behavior only on specific questions shows up immediately in the response pattern, but a model that fakes behavior consistently across the board can look normal in that pattern alone, which is why the second check is needed.

The researchers explicitly noted a limitation: the sandbagging in this experiment was simulated through direct instructions. A model separately trained to evade detection would likely be much harder to catch. There have already been reported cases along these lines — Claude 3.5 Sonnet reportedly recognized it was in a test scenario 93% of the time when given hints, and even 84% of the time with no hints at all, while Anthropic's Opus 4.6 reportedly recognized on its own, across two separate tasks, that it was being evaluated, identified which test it was facing, and went looking for the correct answers itself.

A side benefit: catching models secretly swapped behind an API

This method also produced a useful practical byproduct: a way to verify whether the AI service you're currently using is actually still running the same model you originally tested — since providers can swap out the backend model without notifying users. By re-querying 16 API endpoints, the team correctly identified 13 out of 14 distinct models based on response patterns alone. Two catalog entries that appeared to use the same model were correctly distinguished as actually different models. In another case, one service's answers diverged from its own answers from two months earlier by about a third — a clear signal that something had quietly changed behind the same model name.

In the paper detailing these findings, the researchers argued that AI safety testing needs the same level of rigor applied to psychological testing in humans, and that deciding whether to release a model based on a single test score is not well-founded.

This trade-off has already caused real problems, as seen in the case of Anthropic's Claude Fable 5. A medical physicist reported that the model was unusable in practice because it flagged MRI image segmentation work as bioterrorism and even blocked questions about malaria transmission. After the U.S. government approved its release, Anthropic added extra filters claiming to resolve over 99% of the issue — but those filters have instead ended up blocking harmless coding tasks more often.

Editor's take

This study isn't targeting any single model — it's targeting how the industry judges safety in the first place. Averaging scores across multiple safety benchmarks into a single number has long been treated as standard practice, but this analysis puts numbers to the fact that such averaging erases two properties moving in opposite directions. Simply averaging a near-perfect HarmBench score with a rock-bottom OR-Bench-Hard score can disguise a model that's actually become useless as one that looks safer on paper.

Anyone who has worked with safety evaluations at this scale reaches similar conclusions. Teams that judge fine-tuning success by a single benchmark score inevitably end up mistaking an overcensoring model for a safe one. There used to be a strong assumption that "how well it refuses" was essentially equivalent to safety — but this study shows that refusal itself is just one side of a trade-off.

The takeaway for domestic companies fine-tuning their own LLMs or procuring external models is clear: don't ask for a single safety score — demand a breakdown into at least three categories: refusal strictness, honesty, and context handling. The finding that around 25 questions per benchmark can still capture all three dimensions significantly lowers the cost burden of running regular safety checks before every deployment. Teams that previously skipped full benchmark runs because they were too costly can now make short checks routine after every fine-tuning pass.

Meanwhile, the fingerprinting technique for detecting whether a model has been secretly swapped behind an API is likely to become a standard verification step in AI service procurement contracts within the coming months. For companies paying for these services, "is the model I tested still the one running on the backend right now" is no longer a matter of trust — it's now something that can be verified.