Evidence-RL: Towards Evidence-intensive Visual Reasoning

arXiv:2608.080212026-08-07

一种在训练阶段检验'答对了是不是真的看了图'的方法

视觉语言模型（VLM）可能靠语言先验或数据集捷径给出看似正确的答案，而没有真正依赖图像中的相关证据。本文提出CED，一种反事实审计方法：屏蔽掉候选答案所依赖的'证据区域'与其他无关区域，比较两者对答案支持度的下降幅度，并将这一信号并入GRPO强化学习的奖励，形成Evidence-RL。在九个公开基准和四个骨干模型上，Evidence-RL的表现优于此前基于强化学习的后训练方法。

METAL LAB 解读图

Evidence-RL 训练流程结构

证据状态已报告实测结果

采样候选答案策略模型针对图像和问题生成若干候选答案
证据区域与非证据区域的干预将COCO物体框标出的证据区域，以及面积匹配的其他非证据区域中的视觉token，分别替换为其邻近token的均值
计算证据边际比较证据区域干预导致的答案支持度下降幅度与非证据区域的平均下降幅度，得到答案对真实证据的依赖程度分数
并入GRPO奖励将证据边际分数与正确性分数相乘，使同样正确的答案因证据依赖程度不同而获得不同奖励
推理阶段训练完成后的模型在推理时不再执行该审计过程，因此不增加额外的推理开销

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

问题背景：现有的感知感知型强化学习方法（如PAPO、VPPO）只通过整体图像扰动或注意力代理来粗略检验模型是否用到了图像，却没有检验某个具体答案是否因果性地依赖于支持它的局部证据。
方法：CED针对每个采样出的答案，将COCO物体框标出的'证据区域'的视觉token替换为其邻近token的平均值（一种反事实干预），测量答案支持度下降了多少，并与对匹配的非证据区域做同样干预时的下降幅度做比较，得到一个'证据边际'分数。
该证据边际分数与答案正确性一起计入GRPO的奖励函数：同样答对的两条推理路径，若一条依赖真实图像证据、另一条依赖语言先验捷径，会获得差异巨大的奖励——论文中一个案例显示两条都答对'4'的轨迹，因证据依赖程度不同而获得了7倍的奖励差距。
在CountBench、HallusionBench、MMMU等九个基准和Qwen2.5-VL、Qwen3-VL、Qwen3.5-9B等四个骨干模型（另附LLaVA、InternVL3.5作参考）上做了评测，并对比了只对最终答案打分与对整个推理链打分、不同干预方式（均值替换、置零、加噪声）以及候选区域质量不完美时的稳健性。

Figure 1: Left: Task setup with a counting query over two views of the same scene. Upper right: A pretrained VLM gives the same prior-based answer for both views; Evidence-RL inspects the visual evidence and adjusts its answer when the scene changes. Lower right: Causal path decomposition: a candidate answer can be supported by the target visual evidence, by irrelevant context, or by language priors, which motivates CED to test whether the response depends specifically on the evidence path.

Table 1: Signal-validation statistics on counting and presence tasks.
Task	Non-constant raw reward	Zero variance	Mean reward std	Same answer, different reward	Mean negation
Counting	99.5%	24.0%	0.1299	90.0%	0.003
Presence	21.1%	79.0%	0.0497	5.0%	0.777

Figure 2: Overview of Evidence-RL during post-training. A policy samples candidate answers, and CED scores each answer by applying the same feature-space intervention to the proposed Evidence Region and to matched non-evidence Regions. The contrastive support drop yields an evidence margin, which is combined with correctness to form the GRPO reward. The counterfactual audit is used only during training; inference uses the trained VLM normally.

Table 2: Nine-benchmark evaluation using Qwen2.5-VL-7B as the base model for Ours. Backbone, baseline, and benchmark sources are cited in the setup paragraph. Best bold; second-best underlined. Each Δ row reports absolute gain over the corresponding base.
	Grounding	General Reasoning
Model	CountBench	SpatialEval	Hallusion	VLMs AreBlind	FREAK	MathVista	MMBench	MMMU	ScienceQA	Avg
VLM backbones
LLaVA-v1.6-7B	55.60	23.60	51.00	28.23	9.50	22.80	71.30	29.00	57.20	38.69
InternVL3.5-8B	86.90	24.00	69.60	54.92	12.80	51.90	83.40	50.80	89.90	58.25
Qwen2.5-VL-3B	70.71	53.94	65.37	42.46	11.14	52.10	80.40	50.29	74.93	55.70
Qwen2.5-VL-7B	81.82	59.21	68.64	46.44	12.18	63.10	84.73	50.63	83.98	61.19
RL-based methods
VAPO-Thinker-7B	86.90	60.80	71.20	48.62	13.10	51.10	81.80	44.60	82.30	60.05
Δ	+5.10	+1.59	+2.60	+2.18	+0.92	−12.00	−2.90	−6.00	−1.70	−1.14
VPPO-7B	85.90	61.80	68.20	49.35	12.70	67.90	84.90	52.10	88.40	63.47
Δ	+4.10	+2.59	−0.40	+2.91	+0.52	+4.80	+0.20	+1.50	+4.40	+2.28
Perception-R1-7B	84.90	59.70	66.70	45.22	12.20	67.10	81.70	48.10	82.30	60.88
Δ	+3.10	+0.49	−1.90	−1.22	+0.02	+4.00	−3.00	−2.50	−1.70	−0.31
PAPO-G-H	90.90	64.80	69.50	47.92	12.80	69.40	82.80	50.50	85.60	63.80
Δ	+9.10	+5.59	+0.90	+1.48	+0.62	+6.30	−1.90	−0.10	+1.60	+2.61
VLM-R1	74.80	56.40	66.60	44.15	11.00	60.30	79.90	48.60	73.60	57.26
Δ vs. Qwen2.5-VL-3B	+4.10	+2.50	+1.20	+1.69	−0.14	+8.20	−0.50	−1.70	−1.30	+1.56
SophiaVL-R1	82.83	61.38	66.87	47.09	24.18	66.30	87.41	52.18	87.64	63.99
Δ	+1.01	+2.17	−1.77	+0.65	+12.00	+3.20	+2.68	+1.55	+3.66	+2.80
Ours	88.89	63.34	70.08	54.02	26.40	69.40	87.53	57.47	87.08	67.13
Δ	+7.07	+4.13	+1.44	+7.58	+14.22	+6.30	+2.80	+6.84	+3.10	+5.94

Figure 3: A case showing why correctness-only reward is insufficient. Two rollouts from the same GRPO group both answer “4” correctly, but the CED gate g⁡(m) creates a 7× reward gap between a prior-based rollout (g⁡(m)=0.18, R=0.11) and an evidence-grounded rollout (g⁡(m)=1.00, R=0.78), enabling GRPO to select the grounded trajectory.

Table 3: Controlled comparison on Qwen3.5-9B with matched data, router, and compute. Δ is the mean gain over the frozen base; ΔCED is Answer-CED minus correctness-only.
	Grounding	General Reasoning
Method	CountBench	SpatialEval	Hallusion	VLMs AreBlind	FREAK	MathVista	MMBench	MMMU	ScienceQA	Avg	Δ
λ=0 (corr.-only)	89.90	40.45	57.93	72.90	18.73	73.40	79.44	36.00	82.24	61.22	−1.24
VPPO (retrained)	86.87	38.32	58.37	72.97	19.07	72.50	79.62	37.33	81.09	60.68	−1.78
PAPO (retrained)†	86.87	8.59	58.46	69.96	19.40	14.00	44.87	33.22	34.24	41.07	−21.39
Answer-CED	93.90	54.35	78.10	73.10	23.12	83.75	88.58	75.67	93.61	73.80	+11.34
ΔCED	+4.00	+13.90	+20.17	+0.20	+4.39	+10.35	+9.14	+39.67	+11.37	+12.58	–

Figure 4: Training dynamics for Answer-CED and CoT-CED on Qwen3.5-9B. CoT-CED produces larger training-side rewards, but the downstream comparison in Table 5 favors Answer-CED on average; we therefore use Answer-CED as the default variant.

Table 4: Cross-backbone validation of Ours. Each block reports the frozen base, Ours, and absolute Δ against that block’s base. The Qwen3.5-9B block additionally reports 3-seed mean±std, with its mean Δ in the last column.
	Grounding	General Reasoning
Model	CountBench	SpatialEval	Hallusion	VLMs AreBlind	FREAK	MathVista	MMBench	MMMU	ScienceQA	Avg. Δ
Backbone: Qwen2.5-VL-3B
Qwen2.5-VL-3B	70.71	53.94	65.37	42.46	11.14	52.10	80.40	50.29	74.93	–
Ours	70.71	55.56	67.40	49.51	25.51	60.40	82.55	50.57	80.43	–
Δ	+0.00	+1.62	+2.03	+7.05	+14.37	+8.30	+2.15	+0.28	+5.50	+4.59
Backbone: Qwen2.5-VL-7B
Qwen2.5-VL-7B	81.82	59.21	68.64	46.44	12.18	63.10	84.73	50.63	83.98	–
Ours	88.89	63.34	70.08	54.02	26.40	69.40	87.53	57.47	87.08	–
Δ	+7.07	+4.13	+1.44	+7.58	+14.22	+6.30	+2.80	+6.84	+3.10	+5.94
Backbone: Qwen3-VL-8B-Instruct
Qwen3-VL-8B	94.90	65.26	72.86	67.01	28.85	67.54	89.15	57.82	91.99	–
Ours	95.92	67.75	74.30	68.88	31.18	68.04	89.58	58.85	92.82	–
Δ	+1.02	+2.49	+1.44	+1.87	+2.33	+0.50	+0.43	+1.03	+0.83	+1.33
Backbone: Qwen3.5-9B
Qwen3.5-9B	80.80	52.73	47.20	46.44	9.95	80.00	88.03	66.21	90.78	–
Ours	93.90	54.35	78.10	73.10	23.12	83.75	88.58	75.67	93.61	–
Δ	+13.10	+1.62	+30.90	+26.66	+13.17	+3.75	+0.55	+9.46	+2.83	+11.34
Ours (3 seeds)	93.60 ±0.48	56.04 ±0.27	78.35 ±0.21	73.37 ±0.17	23.11 ±0.40	84.40 ±0.50	89.87 ±0.04	75.39 ±0.22	94.46 ±0.13	+11.83

Figure 5: CED probe robustness under proposal perturbation. (a) Replacing the COCO proposal with a random object box as Ωev collapses the evidence margin to near zero across all seven task types, confirming that the signal tracks proposal relevance rather than masking magnitude. (b) IoU-graded degradation on the count-exclusion diagnostic: m¯ decreases monotonically as spatial overlap with the original proposal drops, approaching the random-region floor at lowest IoU.

Table 5: Answer vs. CoT variants on Qwen3.5-9B; Avg. Δ is the mean gain over the base.
	Grounding	General Reasoning
Variant	CountBench	SpatialEval	Hallusion	VLMs AreBlind	FREAK	MathVista	MMBench	MMMU	ScienceQA	Avg. Δ
Answer	93.90	54.35	78.10	73.10	23.12	83.75	88.58	75.67	93.61	+11.34
CoT	92.90	53.80	78.10	69.09	30.35	81.88	88.53	68.23	94.66	+10.60

Figure A.1: CED probe robustness under proposal perturbation on the count-exclusion diagnostic. (a) Mean evidence margin m¯ across IoU bins, with the random-region baseline as a grey dashed reference. (b,c) m¯ under scale and translation perturbations, with the original ±1 SE band in red. Error bars are ±1 SE.

Table A.1: Complete hyperparameter listing. “Paper notation” gives the corresponding symbol in the main text when applicable.
Category	Parameter	Value	Paper notation
RL & optimization
Learning rate	lr	1×10−5	—
KL penalty coefficient	kl_coeff	0.01	—
GRPO group size	group_size	32	—
Training steps	n_steps	2,000	—
Trainable layers	n_trainable_layers	4 (last)	—
Precision	dtype	bfloat16	—
Generation & sampling
Temperature	temperature	1.0	—
Top-p	top_p	0.95	—
Max new tokens (evidence)	max_new_tokens	32	—
Max response tokens (full)	max_response_tokens	64	—
Prompt format	short_evidence_v1	—	—
Reward function (Equation 4)
Response reward weight	alpha_resp	0.70	see note†
Answer reward weight	alpha_ans	0.30	see note†
Response logprob temp.	tau_resp	0.20	—
Answer logprob temp.	tau_ans	1.00	—
Gate temperature	tau_resp (shared)	0.20	τg
Gate baseline at m=0	(derived)	0.50	g⁡(0)
Evidence tie-breaker	evidence_eps	0.10	εtie
Additive evidence weight	evidence_eps (shared)	0.10	λ
Reward clipping	[min, max]	[−1.25, 1.00]	—
Format penalty	—	−0.75	—
Data & evaluation
Rerank candidates	rerank_num_candidates	6	—
non-evidence Regions per sample	K	3	K

Figure A.2: Qualitative examples of same-answer-different-reward under the CED method. All rollouts share the same answer, while CED produces reward spans of 0.50–0.67 according to visual evidence quality.

Table A.2: Methodological comparison with perception-aware VLM RL methods. The Causal audit column indicates whether the method tests causal evidence dependence for a candidate answer.
Method	Signal source	Granularity	Causal audit	Stage
PAPO [36]	Global mask KL	Global	No	Training
VPPO [10]	Attention weights	Token-level	No	Training
Vision-SR1†	Text self-verification	Description-level	No	Training
PaLMR‡	LLM-as-Judge	Trajectory-level	No	Training
SophiaVL-R1 [6]	Thinking-reward RM	Trajectory-level	No	Training
PEARL§	Perception checklist	Sample-level	No	Training
VAPO¶	Trajectory anchoring	Token-level	No	Training
VLM-R1*	Self-verification	Response-level	No	Training
Perception-R1††	Proxy localization	Region-levela	No	Training
CED (Ours)	Counterfactual intervention	Region-level	Yes	Training

Figure A.3: Within-group zero-variance rate across task families. Evidence-intensive tasks show a large reduction from correctness-only reward to our reward; binary-dominated groups remain near high zero-variance because of their two-action structure.

Table A.3: Cross-model evidence sensitivity validation (N=50 per task per model).
Model	Task	Relevant > Random rate	Mean margin
InternVL3.5-8B	Counting	0.68	0.0070
Presence	0.62	0.0737
LLaVA-v1.6-7B	Counting	0.78	0.0074
Presence	0.60	0.1625

Figure A.4: Failure cases on presence tasks: identical negation answers with identical rewards yield zero within-group variance.

Table A.4: Blindfold test results (N=500 per task family).
Condition	Task	Rel. > Rand. rate	Mean Delta
Normal	Counting	0.634	+1.928
Normal	Presence	0.550	+0.780
Blindfold	Counting	0.404	−0.114
Blindfold	Presence	0.448	+0.070

Table A.5: Perturbation-family ablation within the local intervention family, evaluated by logits-JS AUC.
Key mode	Mean repl.	Zero repl.	Gaussian-noise repl.
prompt_last	0.6780	0.6602	0.6299
answer_first	0.6592	0.6215	0.6273
Average	0.6686	0.6409	0.6286

Table A.6: Gaussian-noise profile under prompt_last. Stability drops under logits_only.
Config	Fixed metric	Best metric	Best paired AUC
logits24	0.6299	0.6299	0.7108
logits_only	0.5753	0.5920	0.6407

Table A.7: Evidence weight (λ) sensitivity. λ=0.00 is correctness-only.
λ	Non-const. rate	Reward std	Zero-var. rate
0.00	100%	0.0528	0%
0.05	100%	0.0508	0%
0.10	96%	0.0597	4%
0.15	100%	0.0415	4%
0.20	100%	0.0491	0%

Table A.8: Gate temperature (τg) sensitivity.
τg	Non-const. rate	Reward std	Zero-var. rate
0.10	100%	0.0568	4%
0.20	96%	0.0597	4%
0.30	96%	0.0469	4%
0.50	100%	0.0378	0%

Table A.9: Proposal-perturbation results on count-exclusion (n=41). Higher values indicate stronger evidence dependence; the random row gives the floor without spatial targeting.
Axis	Condition	m¯	Δ	Rel. rate
IoU stratification	[0.7, 1.0]	0.501	2.032	0.732
[0.5, 0.7)	0.379	1.282	0.683
[0.3, 0.5)	0.341	1.386	0.683
[0.0, 0.3)	0.219	0.555	0.561
Scale	0.50×	0.377	1.113	0.732
0.75×	0.481	1.750	0.732
1.25×	0.433	2.230	0.683
1.50×	0.295	2.003	0.634
2.00×	0.382	1.880	0.707
Shift	5%	0.304	1.602	0.634
10%	0.428	1.457	0.707
20%	0.532	1.899	0.732
original proposal (reference)	0.386	1.852	0.683
random region (floor)	0.147	0.277	0.537

Table A.10: Unified-condition baseline comparison on the offline rerank dataset (N=1,353).
Method	Reward mean	Signal strength	Signal-corr
Vanilla GRPO	−0.149	N/A	N/A
PAPO-Lite	−0.156	KL =0.024	0.034
CED (Ours)	+0.448	Margin =−0.007	Selective

Table A.11: Unified-condition method comparison on the counting shared-candidate slice: 94 samples with at least two distinct candidates.
Method	Acc.	Strong inf.	rnc↑	flip ↑
LogProb	94.7	75.0	–	–
Correctness-only	97.9	100.0	12.8	4.3
CED (Ours)	97.9	100.0	100.0	76.6
Perception-R1-Lite	97.9	100.0	13.8	5.3
VLM-R1-Lite∗	94.7	75.0	0.0	0.0
VPPO-Lite∗	94.7	75.0	0.0	0.0
PAPO-Lite∗	94.7	75.0	0.0	0.0
∗ No reward-based reranking recovered; falls back to log probability.

Table A.12: CoT-CED vs. Answer-CED: training-end summary over 2,000 steps.
Metric	CoT-CED	Answer-CED
Mean training reward	+0.259	−0.009
Final training correctness	43.4%	34.2%

Table A.13: End-of-training CoT behavior on held-out prompts. CoT-CED collapses toward an object-cue answer format, while Answer-CED preserves natural multi-sentence reasoning.
Metric	Benchmark	CoT-CED	Answer-CED
Mean CoT length (tokens)	Counting	3.6	49.3
Median CoT length (tokens)	Counting	3	47
p90 CoT length (tokens)	Counting	6	78
Mean CoT length (tokens)	HallusionBench	3.8	82.1
Median CoT length (tokens)	HallusionBench	3	61
p90 CoT length (tokens)	HallusionBench	6	173
Image-referential opener rate	Counting	1.0%	79.8%
Image-referential opener rate	HallusionBench	5.2%	64.2%
Unique-token ratio	Counting	99.1%	75.8%
Unique-token ratio	HallusionBench	96.8%	68.1%

Table A.14: Representative side-by-side generations. Both variants produce the correct final answer, but CoT-CED reduces the chain to an object cue, while Answer-CED keeps an explicit visual check.
Prompt	CoT-CED	Answer-CED
How many beds are there in the image?	The beds / 2	Got it, let’s look at the image. There’s a loft bed with a lower bed. The upper bunk is one, the lower bed is another. So that’s two beds.
How many weights are there in the image?	The weights / Two	Got it, let’s look at the image. The man is holding two dumbbells, one in each hand. There’s one in the left hand and one in the right hand. So that’s two weights.

研究结果

表3：在Qwen3.5-9B上使用相同数据、路由器和2000步训练配置的受控对比中，Answer-CED相比仅用正确性的奖励平均提升12.58分，相比重新训练的VPPO提升13.12分。
表2：以Qwen2.5-VL-7B为基座，在九个基准上Evidence-RL在对比的强化学习后训练方法中取得最高平均分和最大平均提升，且是唯一在全部九个基准上都没有出现负提升的方法。
表4：在两个模型系列的四个骨干模型（Qwen2.5-VL-3B/7B、Qwen3-VL-8B、Qwen3.5-9B）上，该方法在每个骨干上都取得正的平均提升，且在全部36个基准×骨干组合中都没有出现负提升。
图5：将基于COCO的候选区域替换为随机物体框后，证据边际均值从0.268骤降至0.015，'证据区域优于随机区域'的比例也从0.636降到0.487（接近随机水平），证明该信号确实追踪的是区域相关性而非单纯的屏蔽幅度。
在ARC、OpenBookQA、CommonsenseQA等八个纯文本基准上，Qwen3.5-9B的Answer-CED模型平均准确率变化仅为-0.27个百分点，单个基准的最大下降不超过1.5个百分点。

可应用场景

为容易出现幻觉或捷径推理的视觉问答、计数、空间推理类VLM设计强化学习后训练的奖励信号
在正确率无法区分'蒙对'和'依据证据答对'的场景下，用于训练数据质量诊断或筛选
需要仅凭低成本的弱物体检测框（如COCO风格标注）、无需针对具体问题的证据标注即可构建视觉依据审计信号的场景

局限与待验证事项

实验仅限于使用COCO风格物体框做以物体为中心的区域干预，扩展到属性级、关系级或文本片段级证据的方案只是概念性提出，未单独验证。
对于是/否这类二元判断的存在性问题，由于答案空间只有两种可能，组内奖励的方差本身就受限，论文报告了这一局限。
反事实审计仅在训练阶段使用，推理阶段模型正常运行、不再执行该检验，因此部署阶段是否持续保持证据依赖性并未在本文中直接测量。
对候选区域的缩放和平移扰动实验只在测试范围内未观察到系统性性能下降，超出该范围的稳健性尚未验证。
在图像为空白占位符的ScienceQA样本上，CED模型给出'无法从图像判断'的回答被判为错误，而依赖语言先验蒙对的基线模型反而被判为正确，说明该方法与部分基准的评分方式存在冲突。

为什么重要

仅凭正确率无法区分模型是'蒙对'还是'真的看图'得出答案，而这种方法提供了一种在训练阶段就能奖励两者差异的实用手段。对于构建容易出现幻觉或走捷径问题的视觉问答、计数、空间推理系统的团队，这提供了一种可参考的奖励设计思路。

本文术语

VLM（视觉语言模型） · 同时接收图像和文本输入并生成回答的模型
反事实干预 · 人为屏蔽某个区域的信息，观察模型输出如何变化的实验方法
GRPO · 一种强化学习方法，对同一问题采样多个答案，并在组内做相对排序打分
证据边际（evidence margin） · 屏蔽证据区域导致的支持度下降幅度，减去屏蔽其他区域导致的平均下降幅度
Answer-CED 与 CoT-CED · 两种变体：一种只对最终答案片段打分，另一种对整个推理链打分

论文原文摘要（英文）

Vision-Language Models (VLMs) should answer from concrete image evidence rather than language priors, dataset shortcuts, or irrelevant visual context. Existing perception-aware post-training methods encourage image use through global perturbations or attention proxies, but they do not test whether a sampled answer causally depends on the local evidence that supports it. We propose Counterfactual Evidence Disentanglement (CED), a training-time evidence audit for VLM grounding. For each response, CED neutralizes an object-centric Evidence Region and compares the resulting support drop against matched non-evidence Regions. We combine this signal with answer correctness inside GRPO, rewarding correct answers that rely on the evidence path rather than shortcut or nuisance paths. CED uses weak object-level proposals, requires no question-specific evidence annotations, and adds no inference-time overhead. Across nine public benchmarks and four backbones, CED outperforms prior RL-based post-training methods, with targeted analyses verifying its object-centric signal.

作者 · Haojie Huang

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Haojie Huang et al., arXiv:2608.08021, arxiv-nonexclusive