What AI Red-Team Evaluations Can and Cannot Prove

arXiv:2607.217352026-07-22

There's a calculable limit to how much a 'no harm found' AI safety test can actually prove

This paper works out, in closed-form math, exactly how much evidence a red-team safety test can provide when it finds zero harmful outputs. For common harm categories (around a 1 percent rate), existing public benchmarks already provide meaningful certification, but for rare categories, no benchmark of any feasible size can certify safety at all past a calculable boundary. The authors argue labs should compute this boundary before running an evaluation, not after.

METAL LAB explanatory visual

How the evidential ceiling decides what a safety test can and can't prove

Evidence statusMeasured results reported

Set up the testDefine the safe vs. unsafe hypothesis pair, sample size n, target improvement ratio r, and acceptable risk threshold tau
Score the clean resultCompute the likelihood ratio for zero harmful outputs to see exactly how much it shifts belief toward safety
Locate the boundary pminTheorem 1: above pmin, a finite feasible sample size can certify safety. Theorem 2: below pmin, no feasible benchmark size can
Check real benchmarks against itSimulate eight suites (AdvBench, HarmBench, SafetyBench, etc.) at 1 percent and 0.0001 percent harm rates to measure their statistical power and false-certification risk
Standardize reportingRequire sample size, clustering info, and confidence intervals to be published together so readers know exactly what claim a result supports

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The paper defines an 'evidential ceiling': the maximum factor by which a single test result can shift belief about whether a model is safe, given a fixed testing budget.
It derives a closed-form formula for how much evidence a clean benchmark result (zero harmful outputs observed) actually carries, letting anyone compute the minimum sample size needed to certify safety at a given harm rate and improvement target.
At a 1 percent harm rate, existing public benchmarks like AdvBench (520 prompts) can already support a tenfold shift in belief toward safety, but at a 0.1 percent rate the required sample jumps to 9,203 prompts, and at 0.01 percent to 92,096.
Below a specific boundary harm rate (about 1.4x10^-5 under one set of assumptions: Nmax=10^5, r=0.5, tau=0.5), the paper mathematically proves that no benchmark of any feasible size can turn a clean result into meaningful evidence of safety.
Monte Carlo simulation (8,000 iterations per point) of eight evaluation suites (AdvBench, HarmBench, SafetyBench, XSTest, etc.) found that at a 1 percent harm rate, none reached 80 percent statistical power, while most suites were adequately powered at an 8 percent harm rate.

Figure 1: The evidential ceiling and the two evidence regimes. (a) The evidence channel. Both ceilings are fixed by the channel and the budget; by the data processing inequality no rescoring or aggregation can raise either. The exculpatory ceiling Ceil− governs certification, and for a passive benchmark it is attained at k=0. (b) Evidence contributed by a single result, in bits, both quantities scored against the same hypothesis pair (H0: p=rpu versus H1: p=pu) at r=0.5. Solid curves are a clean sheet, dashed curves one observed harmful output. Open circles mark the crossing rate of Corollary 1. To the right of a circle the clean sheet is the stronger evidence; to the left it is not. As p falls the clean sheet carries vanishing evidence while the single harm converges to log2⁡(1/r)=1 bit, independent of n.

Table 1: Two regimes, one hypothesis pair. Evidence in bits carried by a clean benchmark and by one observed harmful output, both scored against H0: p=rpu versus H1: p=pu, at n=520 and r=0.5. The ordering reverses at p×=1.33×10−3 (Corollary 1). Neither observation is universally the stronger.
pu	10−2	5×10−3	10−3	10−4	10−5
\|log2⁡Λ0\|, clean sheet	3.779	1.882	0.375	0.038	0.004
\|log2⁡Λ1\|, one observed harm	2.772	0.882	0.625	0.963	0.996
ratio, harm to clean sheet	0.73	0.47	1.7	25.7	265.6

Figure 2: Statistical power across eight evaluation suites. Power to detect a fifty percent reduction in harm rate, from 8,000 Monte Carlo iterations per operating point at α=0.05. Dashed lines mark the frontier operating point (p=0.01) and a high-frequency category (p=0.08). At p=0.08 most suites are adequately powered; at p=0.01 none reaches 80 percent.

Table 2: Discrimination, not attack success, determines evidential worth. Evidence in bits carried by a null result, for illustrative hypothesis-conditioned rates. The values are chosen to display the structure and are not empirical estimates for any published procedure. Rows three and five have the same elicitation rate under H1 and differ by two orders of magnitude in the evidence a null result carries.
Procedure	Trial unit	n	q1	q0	bits
Passive benchmark, AdvBench-sized	prompt	520	10−3	5×10−4	0.38
Passive benchmark, at Nmax	prompt	105	10−3	5×10−4	72.19
Adaptive campaign, discriminating	campaign	1	0.90	0.10	3.17
Adaptive campaign, discriminating	campaign	5	0.90	0.10	15.85
Adaptive campaign, non-discriminating	campaign	5	0.90	0.85	2.92

Figure 3: False certification and the boundary. (a) False certification against benchmark size for four harm rates, with the range of current public benchmarks shaded. (b) The boundary pmin(τ,Nmax,r) against the feasibility ceiling, for three combinations of τ and r. Above a curve, Theorem 1 gives a finite sufficient n; in the shaded region below, Theorem 2 applies and no feasible benchmark certifies.

Table 3: False-certification probability across the operating surface. The probability that a model with true harm rate p produces zero harmful outputs in n approximately independent trials. Values above roughly five percent mark operating points at which a clean result is not informative evidence. Current public benchmarks occupy the leftmost three columns, where they are adequate at p=10−2 and inadequate below it.
P⁡(k=0∣n,p), %	n=250	n=520	n=2,100	n=5,000	n=104	n=105
p=10−2	8.1	0.5	0.0	0.0	0.0	0.0
p=10−3	77.9	59.4	12.2	0.7	0.0	0.0
p=10−4	97.5	94.9	81.1	60.7	36.8	0.0
p=10−5	99.8	99.5	97.9	95.1	90.5	36.8

Figure 4: Benchmark and deployment prompt distributions. Joint UMAP projection [25] of AdvBench, HarmBench, and a 9,089-query sample of LMSYS-Chat-1M under sentence-transformer embeddings. The benchmarks occupy a narrow region measurably separated from ordinary deployment traffic. This does not bound the distance to the adversarial component of deployment, which is the component catastrophic-risk claims concern.

Table 4: The claims ladder. Required sample size, at r=0.5 and one-sided α=0.05, for each level of claim at four harm rates. Current public benchmarks (n≤2,100) support every claim in the table at p=10−2, the weakest two at p=10−3, and only the upper bound below that. The final row applies Theorem 2 at Nmax=105.
Claim the evaluation licenses	p=10−2	p=10−3	p=10−4	p=10−5
Harm rate bounded above at 95% confidence	any n	any n	any n	any n
Belief shifts toward safety by 2× (τ=0.5)	138	1,385	13,862	138,628
False certification held below 5%	298	2,994	29,956	299,572
Belief shifts by 10× (τ=0.1)	457	4,602	46,048	460,514
Belief shifts by 100× (τ=0.01)	914	9,203	92,096	921,027
Feasible at Nmax=105?	yes	yes	partly	no

Table 5: Minimum reporting template for a red-team null result. To be completed once per harm category, per model, before the result enters a safety case.
Field	Symbol	Requirement
Harm category	—	Named, with the threat model it operationalizes
Elicitation procedure	—	Passive corpus, adaptive campaign, or other; stated
Trial unit	—	Prompt, campaign, or replication; campaigns are not prompts
Nominal sample size	n	Exact count of independent trials in the stated unit
Event count	k	Exact integer, at the same analysis unit as n
Generations per prompt	m	Exact; state if greater than one
Intra-cluster correlation	ρ	Estimated from the run, or an upper bound
Effective sample size	neff	n/[1+(m−1)ρ]
Upper confidence bound	—	Exact Clopper-Pearson at 95% on the nominal (n,k)
Clustering-adjusted bound	—	Beta-binomial or GEE; not Clopper-Pearson on neff
Minimum detectable effect	δmin	Prespecified, not chosen post hoc
Observed power	1−β	Against δmin, with sidedness stated
Evidentiary threshold	τ	With the loss ratio L and prior odds O0 that fix it
Elicitation rate under H1	q1	Measured against a positive control; state the control
Elicitation rate under H0	q0	Stated or bounded; q1 alone is not sufficient
Per-trial discrimination	κ	\|log2⁡[(1−q1)/(1−q0)]\|, in bits
Achieved likelihood ratio	Λ0	[(1−q1)/(1−q0)]n exactly, reported as a number
Claim licensed	—	The highest row of Table 4 the design supports
Distributional coverage	—	Distance to each deployment mixture component
Semantic consistency	σ	Across prompt transformation types
Out-of-distribution gap	—	Attack success on held-out attack families
Position relative to boundary	—	Whether p is suspected below pmin(τ,Nmax,r)

Findings

At a 1 percent harm rate with 520 prompts (AdvBench-sized), the paper calculates that a clean result carries 1.4 times more evidence than a single observed harmful output, with the crossing point between the two occurring at a harm rate of about 1.33x10^-3.
Power simulations across eight evaluation suites (AdvBench, HarmBench, SafetyBench, XSTest, StrongREJECT, etc.) showed power ranging from 14.9 percent (XSTest) to 59.3 percent (SafetyBench) at a 1 percent harm rate, all below the 80 percent adequacy threshold.
At a 0.1 percent harm rate, StrongREJECT (313 prompts) has a 73.1 percent chance of falsely certifying an unsafe model as safe, and AdvBench has a 59.4 percent chance.
AdvBench and HarmBench prompts sit 3.1 to 3.5 times farther from real user conversation data (a 9,089-query sample from LMSYS-Chat-1M) in embedding space than the benchmarks are from each other, showing they don't represent ordinary deployment traffic.
Reviewing nine public frontier model safety disclosures, the authors found only one endpoint reported both an exact numerator and denominator for a binary harm rate, and none reported the dependence structure needed to interpret it.

Where it can be used

AI labs could use this framework to check, before running an evaluation, whether a benchmark of a given size can even in principle certify safety for a specific harm category at a stated confidence level.
Auditors or regulators could use the same math as a checklist to interpret what a 'zero harmful outputs observed' line in a system card actually implies statistically.
Teams writing safety reports could adopt the paper's minimum reporting template (sample size, clustering structure, confidence interval) to make their claims verifiable.

Limits and open work

The core mathematical inequality behind both theorems dates back to 1983; the paper's contribution is applying it to evaluation design, not the math itself.
The harm-rate estimates used in the power analysis are described as defensible rather than authoritative, since published estimates for frontier models are sparse; only the qualitative pattern (adequate at 1 percent, inadequate at 0.0001 percent) is claimed to be robust.
The distributional comparison only measured distance from ordinary user conversation data (LMSYS-Chat-1M) and explicitly does not measure distance to real adversarial attack traffic.
Both theorems assume approximately independent trials; adaptive red-teaming where a model detects it is being tested could change the underlying statistics, and the paper leaves open how much this would shift the boundary.
The illustrative numbers in Table 2 are meant to show structure, not to represent measured rates from any actual published evaluation procedure.

Why it matters

This gives AI labs, auditors, and regulators a way to check whether a 'no harmful outputs found' claim in a safety report or system card is actually statistically meaningful, rather than just assuming bigger test sets always help. It also tells the field when to stop asking for larger benchmarks and instead demand different kinds of evidence for extremely rare harm categories.

Terms in this paper

evidential ceiling · the maximum factor by which a single test result can shift belief about safety, given a fixed testing budget
null result · a test outcome where none of the prompts produced a harmful response
likelihood ratio (Λ) · a number showing how much better one hypothesis (unsafe vs. safe) explains the observed result than the other
Clopper-Pearson interval · a precise statistical method for estimating the plausible range of a true rate from an observed count of events
design effect · a correction factor accounting for how test prompts drawn from similar templates are not fully independent, shrinking the effective sample size

Original abstract (English)

Red-team evaluations of AI models support some claims and not others, and the boundary between the two is calculable rather than merely a matter of judgment. We define the evidential ceiling of an evaluation as the largest factor by which one result can move belief under a fixed testing budget, derive it in closed form for the benchmark null result, and use it to locate that boundary exactly. We find that above a calculable harm rate, a benchmark of modest size certifies a category to a stated e

Authors · Bandana Kaur

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Bandana Kaur et al., arXiv:2607.21735, CC BY 4.0