What AI Red-Team Evaluations Can and Cannot Prove

arXiv:2607.217352026-07-22

AI红队测试说'没发现问题'时，这句话到底能证明多少,是可以算出来的

这篇论文用封闭形式的数学公式,精确计算出一次红队安全测试在零有害输出的情况下究竟能提供多少证据力。对于常见的危害类别(发生率约1%),现有公开基准测试已经足以支撑有意义的安全认证;但对于罕见类别,一旦低于某个可计算的临界发生率,无论基准测试规模多大都无法证明安全性。作者主张实验室应在运行评估之前先算出这个边界,而不是事后才发现问题。

METAL LAB 解读图

证据天花板如何决定一次安全测试能证明什么、不能证明什么

证据状态已报告实测结果

设定测试条件确定安全与不安全的假设对、样本量n、目标改进比率r,以及可接受的风险阈值tau
计算干净结果的证据力计算零有害输出情况下的似然比,量化这一结果究竟把信念向安全方向推动了多少
定位边界pmin定理1:发生率高于pmin时,存在有限的可行样本量能够认证安全;定理2:低于pmin时,任何可行规模的基准测试都做不到
对照真实基准测试对AdvBench、HarmBench、SafetyBench等八个套件在1%和0.0001%发生率下模拟检验力和误认证概率
统一报告方式要求同时公布样本量、聚类信息和置信区间,让读者清楚一个结果究竟支持什么程度的结论

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

论文提出了证据天花板这一概念,即在固定测试预算下,单次测试结果最多能把人们对模型安全与否的信念改变多少倍。
研究推导出一个闭式公式,用来计算一次干净的基准测试结果(即零有害输出)实际携带多少证据力,从而可以提前算出在给定发生率和目标改进幅度下需要多少样本量才能认证安全。
在1%的危害发生率下,AdvBench这类现有公开基准(520个提示词)已经能支持十倍量级的信念上调,但发生率降到0.1%时所需样本量跃升到9203个,降到0.01%时更是需要92096个。
在某个临界发生率以下(在Nmax=10^5、r=0.5、tau=0.5的假设条件下约为1.4x10^-5),论文用数学证明表明,无论基准测试规模多大,零有害输出的结果都无法构成有意义的安全证据。
对AdvBench、HarmBench、SafetyBench、XSTest等八个评估套件进行了8000次蒙特卡洛模拟,结果显示在1%的危害发生率下,没有一个套件的统计检验力达到80%,而在8%的发生率下大多数套件的检验力是足够的。

Figure 1: The evidential ceiling and the two evidence regimes. (a) The evidence channel. Both ceilings are fixed by the channel and the budget; by the data processing inequality no rescoring or aggregation can raise either. The exculpatory ceiling Ceil− governs certification, and for a passive benchmark it is attained at k=0. (b) Evidence contributed by a single result, in bits, both quantities scored against the same hypothesis pair (H0: p=rpu versus H1: p=pu) at r=0.5. Solid curves are a clean sheet, dashed curves one observed harmful output. Open circles mark the crossing rate of Corollary 1. To the right of a circle the clean sheet is the stronger evidence; to the left it is not. As p falls the clean sheet carries vanishing evidence while the single harm converges to log2⁡(1/r)=1 bit, independent of n.

Table 1: Two regimes, one hypothesis pair. Evidence in bits carried by a clean benchmark and by one observed harmful output, both scored against H0: p=rpu versus H1: p=pu, at n=520 and r=0.5. The ordering reverses at p×=1.33×10−3 (Corollary 1). Neither observation is universally the stronger.
pu	10−2	5×10−3	10−3	10−4	10−5
\|log2⁡Λ0\|, clean sheet	3.779	1.882	0.375	0.038	0.004
\|log2⁡Λ1\|, one observed harm	2.772	0.882	0.625	0.963	0.996
ratio, harm to clean sheet	0.73	0.47	1.7	25.7	265.6

Figure 2: Statistical power across eight evaluation suites. Power to detect a fifty percent reduction in harm rate, from 8,000 Monte Carlo iterations per operating point at α=0.05. Dashed lines mark the frontier operating point (p=0.01) and a high-frequency category (p=0.08). At p=0.08 most suites are adequately powered; at p=0.01 none reaches 80 percent.

Table 2: Discrimination, not attack success, determines evidential worth. Evidence in bits carried by a null result, for illustrative hypothesis-conditioned rates. The values are chosen to display the structure and are not empirical estimates for any published procedure. Rows three and five have the same elicitation rate under H1 and differ by two orders of magnitude in the evidence a null result carries.
Procedure	Trial unit	n	q1	q0	bits
Passive benchmark, AdvBench-sized	prompt	520	10−3	5×10−4	0.38
Passive benchmark, at Nmax	prompt	105	10−3	5×10−4	72.19
Adaptive campaign, discriminating	campaign	1	0.90	0.10	3.17
Adaptive campaign, discriminating	campaign	5	0.90	0.10	15.85
Adaptive campaign, non-discriminating	campaign	5	0.90	0.85	2.92

Figure 3: False certification and the boundary. (a) False certification against benchmark size for four harm rates, with the range of current public benchmarks shaded. (b) The boundary pmin(τ,Nmax,r) against the feasibility ceiling, for three combinations of τ and r. Above a curve, Theorem 1 gives a finite sufficient n; in the shaded region below, Theorem 2 applies and no feasible benchmark certifies.

Table 3: False-certification probability across the operating surface. The probability that a model with true harm rate p produces zero harmful outputs in n approximately independent trials. Values above roughly five percent mark operating points at which a clean result is not informative evidence. Current public benchmarks occupy the leftmost three columns, where they are adequate at p=10−2 and inadequate below it.
P⁡(k=0∣n,p), %	n=250	n=520	n=2,100	n=5,000	n=104	n=105
p=10−2	8.1	0.5	0.0	0.0	0.0	0.0
p=10−3	77.9	59.4	12.2	0.7	0.0	0.0
p=10−4	97.5	94.9	81.1	60.7	36.8	0.0
p=10−5	99.8	99.5	97.9	95.1	90.5	36.8

Figure 4: Benchmark and deployment prompt distributions. Joint UMAP projection [25] of AdvBench, HarmBench, and a 9,089-query sample of LMSYS-Chat-1M under sentence-transformer embeddings. The benchmarks occupy a narrow region measurably separated from ordinary deployment traffic. This does not bound the distance to the adversarial component of deployment, which is the component catastrophic-risk claims concern.

Table 4: The claims ladder. Required sample size, at r=0.5 and one-sided α=0.05, for each level of claim at four harm rates. Current public benchmarks (n≤2,100) support every claim in the table at p=10−2, the weakest two at p=10−3, and only the upper bound below that. The final row applies Theorem 2 at Nmax=105.
Claim the evaluation licenses	p=10−2	p=10−3	p=10−4	p=10−5
Harm rate bounded above at 95% confidence	any n	any n	any n	any n
Belief shifts toward safety by 2× (τ=0.5)	138	1,385	13,862	138,628
False certification held below 5%	298	2,994	29,956	299,572
Belief shifts by 10× (τ=0.1)	457	4,602	46,048	460,514
Belief shifts by 100× (τ=0.01)	914	9,203	92,096	921,027
Feasible at Nmax=105?	yes	yes	partly	no

Table 5: Minimum reporting template for a red-team null result. To be completed once per harm category, per model, before the result enters a safety case.
Field	Symbol	Requirement
Harm category	—	Named, with the threat model it operationalizes
Elicitation procedure	—	Passive corpus, adaptive campaign, or other; stated
Trial unit	—	Prompt, campaign, or replication; campaigns are not prompts
Nominal sample size	n	Exact count of independent trials in the stated unit
Event count	k	Exact integer, at the same analysis unit as n
Generations per prompt	m	Exact; state if greater than one
Intra-cluster correlation	ρ	Estimated from the run, or an upper bound
Effective sample size	neff	n/[1+(m−1)ρ]
Upper confidence bound	—	Exact Clopper-Pearson at 95% on the nominal (n,k)
Clustering-adjusted bound	—	Beta-binomial or GEE; not Clopper-Pearson on neff
Minimum detectable effect	δmin	Prespecified, not chosen post hoc
Observed power	1−β	Against δmin, with sidedness stated
Evidentiary threshold	τ	With the loss ratio L and prior odds O0 that fix it
Elicitation rate under H1	q1	Measured against a positive control; state the control
Elicitation rate under H0	q0	Stated or bounded; q1 alone is not sufficient
Per-trial discrimination	κ	\|log2⁡[(1−q1)/(1−q0)]\|, in bits
Achieved likelihood ratio	Λ0	[(1−q1)/(1−q0)]n exactly, reported as a number
Claim licensed	—	The highest row of Table 4 the design supports
Distributional coverage	—	Distance to each deployment mixture component
Semantic consistency	σ	Across prompt transformation types
Out-of-distribution gap	—	Attack success on held-out attack families
Position relative to boundary	—	Whether p is suspected below pmin(τ,Nmax,r)

研究结果

在1%危害发生率、520个提示词(相当于AdvBench规模)的条件下,论文计算出一次干净结果携带的证据力是单次观察到有害输出的1.4倍,两者证据力相等的临界发生率约为1.33x10^-3。
对八个评估套件(AdvBench、HarmBench、SafetyBench、XSTest、StrongREJECT等)的检验力模拟显示,在1%危害发生率下检验力介于14.9%(XSTest)到59.3%(SafetyBench)之间,均未达到80%的充分标准。
在0.1%的危害发生率下,StrongREJECT(313个提示词)将实际不安全的模型误判为安全的概率高达73.1%,AdvBench为59.4%。
AdvBench和HarmBench的提示词在嵌入空间中与真实用户对话数据(取自LMSYS-Chat-1M的9089条查询样本)的距离,比基准测试之间彼此的距离还要远3.1到3.5倍,说明这些基准测试并不能代表普通的部署流量。
对九份公开的前沿模型安全披露文件进行审查后发现,针对二元危害率,只有一处同时报告了精确的分子(事件数)和分母(总试验数),且没有一份报告了解释所需的依赖结构信息。

可应用场景

AI实验室可以在运行评估之前,用这一框架计算某个规模的基准测试是否原则上能够认证特定危害类别在给定置信水平下的安全性。
审计机构或监管者可以用同样的数学方法作为核查清单,解读系统卡片中'未观察到有害输出'这句话在统计上究竟意味着什么。
撰写安全报告的团队可以参照论文提出的最低报告模板(样本量、聚类结构、置信区间),让安全声明变得可核查。

局限与待验证事项

支撑两条定理的核心数学不等式早在1983年就已提出,论文的贡献在于将其应用到评估设计上,而非数学本身的创新。
用于检验力分析的危害发生率估计值被作者称为合理但非权威的估计,因为前沿模型的公开发生率数据稀少,只有定性结论(1%时充分、0.0001%时不充分)被认为是稳健的。
分布比较仅测量了与普通用户对话数据(LMSYS-Chat-1M)的距离,明确没有测量与真实对抗性攻击流量之间的距离。
两条定理都假设试验之间近似独立;如果模型能察觉自己正被测试从而采取适应性策略,可能改变底层统计结构,论文将这一边界会如何变化留作未解决的问题。
表2中的示例数值仅用于展示结构,并非任何已发布评估程序的实测结果。

为什么重要

这项研究给AI实验室、审计机构和监管者提供了一种方法,用来判断安全报告或系统卡片中'未发现有害输出'这句话在统计上到底意味着什么,而不是想当然地认为基准测试规模越大越好。它也告诉业界,对于极其罕见的危害类别,应该何时停止要求更大的基准测试,转而要求其他类型的证据。

本文术语

证据天花板(evidential ceiling) · 在固定测试预算下,单次测试结果最多能改变人们对安全性信念的倍数上限
零结果(null result) · 测试的所有提示词中没有一个产生有害回应的结果
似然比(likelihood ratio, Λ) · 衡量观察到的结果更支持不安全假设还是安全假设的数值比
克洛珀-皮尔逊区间(Clopper-Pearson interval) · 根据观察到的事件次数,精确估计真实发生率可能范围的统计方法
设计效应(design effect) · 当测试提示词并非完全独立、而是来自相似模板派生而来时,用来校正有效样本量被高估程度的系数

论文原文摘要（英文）

Red-team evaluations of AI models support some claims and not others, and the boundary between the two is calculable rather than merely a matter of judgment. We define the evidential ceiling of an evaluation as the largest factor by which one result can move belief under a fixed testing budget, derive it in closed form for the benchmark null result, and use it to locate that boundary exactly. We find that above a calculable harm rate, a benchmark of modest size certifies a category to a stated e

作者 · Bandana Kaur

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Bandana Kaur et al., arXiv:2607.21735, CC BY 4.0