每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

TESTNAV: Pareto-Guided Search for Compositional Robustness Testing

arXiv:2608.198822026-08-21

测试AI模型面对多种叠加干扰时不必穷举所有组合,也能找出真正危险的失败案例

当亮度变化、运动模糊等多种干扰同时出现在同一张图片或文本中时,可能的组合数量会爆炸式增长,而且很多组合会让输入变得极不真实,没有测试价值。TESTNAV把稳健性测试变成一个双目标优化问题:既要让模型性能大幅下降,又要让被干扰的输入尽量保持和原始输入相似,并用进化算法NSGA-II高效找出这些平衡点组合。在覆盖图像、文本和代码的四个基准测试中,TESTNAV只需检查全部组合空间的35.8%到89.3%,速度比现有搜索方法最多快2.15倍,且结果一致。

他们做了什么

  1. 现实中的输入往往同时受到多种干扰影响(比如亮度变化加运动模糊),但穷举所有组合会导致组合数量爆炸式增长——四种干扰类型、每种六个强度等级就有1296种组合,而其中很多组合会让输入变得不真实,诊断价值有限
  2. TESTNAV把稳健性测试重新定义为双目标优化问题:一是让干扰组合尽可能拉低模型性能,二是用SSIM、KID(图像)或chrF、BERT-F1(文本与代码)等针对不同数据类型的相似度指标,确保被干扰的输入仍然贴近原始输入
  3. TESTNAV使用进化算法NSGA-II来高效逼近这两个目标之间的最优平衡点集合(帕累托前沿),而不需要评估全部组合
  4. 研究者在Tiny-ImageNet、QQP、HumanEval、MBPP四个基准上穷举评估全部1296种组合建立真实基准后发现,TESTNAV只需检查35.8%至89.3%的组合空间,就能以最多快2.15倍的速度得到与穷举搜索相同的最优组合集合
  5. 只优化性能下降或只优化输入相似度的单目标搜索,以及现有基于神经网络内部激活情况的覆盖率指标,表现都不如TESTNAV,说明必须同时兼顾两个目标才能找到真正有价值的失败案例
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 0
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 1
Table 1: Benchmarks, datasets and models, perturbations, fidelity metrics ϕ, task-performance metrics ψ, and clean-set performance ψ⁡(𝒟).
Dataset 𝒟SizeModelPerturbationsFidelity ϕTask ψClean ψ⁡(𝒟)
Tiny-ImageNet10,000CaiT-S36speckle noise, glass blur, brightness, pixelateKID, SSIMAccuracy86.7%
QQP1,000RoBERTa-basesynonym, typo, contraction, punctuationBERT-F1, chrFAccuracy91.2%
HumanEval164CodeGen-2B-monobutterfingers, char case, whitespace, newlineBERT-F1, chrFRP5@123.2%
MBPP974CodeGen-2B-monobutterfingers, char case, whitespace, newlineBERT-F1, chrFRP5@131.9%
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 2
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 3
Table 2: AUC-Recall@𝒫∗ for TestNav and search baselines across all benchmarks and fidelity metrics. Bold indicates the best score and underline indicates the second-best score per column.
MethodSSIMKIDchrFBERT-F1chrFBERT-F1chrFBERT-F1
TestNav0.7040.6860.6460.6510.7290.7300.6970.690
Greedy Search0.5570.4640.6380.7820.7420.7830.6810.751
Genetic Algorithm0.5400.5520.6930.7070.7530.7670.6400.631
Random Search0.5100.5030.5260.5140.5020.5220.5140.490
Figure 4: Input-level metrics versus TestNav on . Curves show Recall@𝒫∗ over unique configurations (top) and evaluation budget (bottom), using SSIM (left) and KID-derived fidelity (right).
Figure 4: Input-level metrics versus TestNav on . Curves show Recall@𝒫∗ over unique configurations (top) and evaluation budget (bottom), using SSIM (left) and KID-derived fidelity (right).
Figure 7: 4D voxel visualisations of 𝒫∗ for non-vision benchmarks using chrF as the fidelity metric.
Figure 7: 4D voxel visualisations of 𝒫∗ for non-vision benchmarks using chrF as the fidelity metric.
Table 3: Pareto-front composition per metric pair for .
Metric pair𝒫∗Θ1Θ2Θ3Θ4
(δ,ρSSIM)2791242
(δ,ρKID)2747142
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 6
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 7

为什么重要

对于自动驾驶、医学影像等对可靠性要求极高的系统来说,穷举测试所有可能的多重干扰组合在计算上是不现实的。TESTNAV提供了一种在有限测试预算内、高效找出真正有意义的失败案例的方法,为模型测试团队设计更可靠的稳健性评估流程提供了实用参考。

Figure 8: 4D voxel visualisations of 𝒫∗ for non-vision benchmarks using BERT-F1 as the fidelity metric.
Figure 8: 4D voxel visualisations of 𝒫∗ for non-vision benchmarks using BERT-F1 as the fidelity metric.
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 9

本文术语

  • 帕累托前沿(Pareto front) · 在两个互相制约的目标之间,无法再同时改善两者的一组最优平衡点
  • NSGA-II · 一种用于同时优化多个相互竞争目标的进化算法
  • SSIM/KID · 衡量被干扰图像与原始图像在结构或统计上相似程度的指标
  • chrF/BERT-F1 · 衡量被干扰文本或代码与原始文本相似程度的指标
  • Recall@P* · 衡量搜索过程中已经找到了多少比例真正最优组合的指标
TESTNAV: Pareto-Guided Search for Compositional Robustness Testing figure 10

论文原文摘要（英文）

Deep learning models remain vulnerable to real-world input perturbations, especially when multiple corruptions co-occur in the same input (e.g., brightness shifts and motion blur). Compositional testing reveals these interaction effects but introduces two challenges: combinatorial growth of the perturbation space as dimensions and severity levels increase, and uneven diagnostic value-many combinations yield unrealistically degraded inputs with limited practical relevance. We present TESTNAV, 1 a Pareto-guided robustness testing framework for efficiently exploring discrete, compositional perturbation spaces when only a limited number of perturbation configurations can be evaluated. TESTNAV prioritises severe yet realistic failures by formulating robustness testing as bi-objective optimisation: maximise performance degradation while preserving input fidelity measured by modality-specific metrics (e.g., SSIM and KID for vision; chrF and BERT-F1 for language and code). It uses NSGA-II to approximate the bi-objective Pareto front. Across four benchmarks spanning vision, natural language, and code generation, TESTNAV recovers Pareto fronts up to 2.15x faster than search-based baselines, using 35.8%-89.3% of the discrete perturbation space defined by four perturbation dimensions with six levels each.

作者 · Arooj Arif, Tobias Hartung, Elena Botoeva, Alexandros Koliousis

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Arooj Arif et al., arXiv:2608.19882, CC BY 4.0