The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
arXiv:2608.062702026-08-05
会放大看图的AI工具,多数情况下其实并没有真正靠看到的内容来改变答案
研究检验了一类会主动裁剪放大图像来辅助推理的多模态AI模型,看看它们截取的图像片段是否真的因果性地影响了最终答案。作者设计了策略级、轨迹级、步骤级三层干预实验,在六个模型和五个细粒度感知基准上发现:准确率的提升大多集中在一小部分“校准良好”的样本上,其余情况要么是调用了工具却没真正看图(Calling Without Looking),要么是看了图但调用节奏毫无规划(Looking Without Planning)。作者将这一现象称为“视觉工具使用的幻觉”。
METAL LAB 解读图
视觉工具使用的三层因果审计结构
证据状态已报告实测结果
策略级干预比较使用工具的模型与直接推理的模型的准确率,衡量拥有工具带来的整体效果
轨迹级干预在推理过程中污染所有返回的图像,观察准确率下降程度,衡量整体观察信息的作用
步骤级干预(VEG)只把某一次返回的图像换成无关的裁剪图像,单独衡量这一次观察对答案置信度的影响
四组诊断分类把每条推理轨迹归入No-call、Calling Without Looking、Looking Without Planning、Calibrated四组,分解准确率提升的来源
将所有推理轨迹分为No-call、Calling Without Looking、Looking Without Planning、Calibrated四组后,再把策略级准确率提升按组分解,发现提升几乎全部来自Calibrated这一小部分。
Figure 1: The paradox of visual tool-use. (a) Our experiments show that, despite far more tokens, tool-use policies yield marginal or no improvements over direct inference. (b) A qualitative failure on a V∗ instance (by Mini-o3): the direct prediction is correct, but tool-augmented inference repeatedly crops irrelevant regions and leads to no valid answer or a wrong answer.
Table 1: Policy-level intervention: tool-use vs. direct inference. Acc reports accuracy (%) under πtool / πdirect; Δ is the policy-level ATE in percentage points (pp) (Eq. 1). VisualProbe combines Easy/Medium/Hard levels.
Model
V∗
HR-Bench-4K
HR-Bench-8K
VisualProbe
MME-RealWorld-Lite
Interface-based
DeepEyes
Acc
83.3/83.3
75.0/76.4
72.0/72.6
42.9/42.9
54.7/54.9
Δ
0.0
−1.4
−0.6
+0.0
−0.2
Pixel Reasoner
Acc
84.8/81.2
74.0/74.4
68.2/64.0
37.3/29.7
64.4/54.4
Δ
+3.6
−0.4
+4.2
+7.6
+10.0
Mini-o3
Acc
87.8/82.3
76.8/72.0
73.2/67.4
55.0/33.7
65.5/53.6
Δ
+5.5
+4.8
+5.8
+21.3
+11.9
Qwen3-VL-8B
Acc
91.1/84.2
83.9/78.9
79.5/75.7
48.6/36.9
51.0/54.7
Δ
+6.9
+5.0
+3.8
+11.7
−3.7
Qwen3-VL-4B
Acc
86.4/82.0
80.1/80.1
75.3/76.9
45.7/38.9
49.0/50.3
Δ
+4.4
0.0
−1.6
+6.8
−1.3
Code-based
Thyme
Acc
83.2/80.2
78.3/74.7
73.0/68.2
46.2/43.7
55.2/48.9
Δ
+3.0
+3.6
+4.8
+2.6
+6.3
(b) The Zoom-Loop Trap
Table 2: Trajectory-level intervention under random-crop. Acc reports accuracy (%) under πtool / the dynamic intervention do(Oi=O~i∀i); Δ is the trajectory-level ATE on accuracy in pp (Eq. 2). The rightmost column reports Hit-MaxT, the fraction of rollouts that hit the tool-call limit.
Model
V∗
HR-Bench-4K
HR-Bench-8K
VisualProbe
MME-RealWorld-Lite
Hit-MaxT (V∗)
Interface-based
DeepEyes
Acc
83.3/83.8
75.0/74.1
72.0/70.4
42.9/44.2
54.7/54.6
0%
Δ
+0.4
−0.9
−1.6
+1.3
−0.1
Pixel Reasoner
Acc
84.8/82.7
74.0/70.8
68.2/66.1
37.3/33.7
64.4/49.3
0%
Δ
−2.1
−3.2
−2.1
−3.6
−15.1
Mini-o3
Acc
87.8/23.6
76.8/20.4
73.2/15.8
55.0/2.4
65.5/15.1
84.8%
Δ
−64.2
−56.4
−57.4
−52.6
−50.4
Qwen3-VL-8B
Acc
91.1/30.4
83.9/46.4
79.5/41.0
48.6/11.2
51.0/29.7
59.2%
Δ
−60.7
−37.5
−38.5
−37.4
−21.3
Qwen3-VL-4B
Acc
86.4/38.2
80.1/59.3
75.3/46.6
45.7/8.8
49.0/34.3
50.3%
Δ
−48.2
−20.8
−28.7
−36.9
−14.7
Code-based
Thyme
Acc
83.2/82.2
78.3/77.9
73.0/74.1
46.2/45.6
55.2/56.2
0%
Δ
−1.0
−0.4
+1.1
−0.6
+1.0
Figure 2: Causal graph for visual tool-use. (a) The single-step causal graph for a single tool-use step over (I,Q,T,O,Y), highlighting the direct path (I,Q)→Y (blue), the observation-mediated path T→O→Y (green) and the action-induced shortcut T→Y (red). (b) A full trajectory composes n such atomic graphs.
Table 3: Step-level intervention on V∗. Marginal Gain (MG) is reported with the original observation (ΔMreal) and with the counterfactual observation (ΔMcf). Visual Evidence Gain (VEG) isolates the contribution of the visual content; we report it pooled over All calls and split by whether the trajectory is ultimately Correct or Incorrect. Distribution shows the fraction of calls that are near-zero (|VEG|<0.01) or substantial (|VEG|>0.1). By saturation conditions VEG on pre-call confidence: Sat-rate is the fraction of calls whose pre-call probability gap gi−1 already exceeds 0.95, and Sat/Non-sat are mean VEG conditional on (non-)saturation. mean Vmax is each trajectory’s best per-call VEG, averaged across trajectories, capturing how much the most useful call in a trajectory contributes on average.
MG (mean)
VEG all-calls (mean)
Distribution
By saturation
Model
ΔMreal
ΔMcf
A
C
I
|VEG|<0.01
|VEG|>0.1
Sat-rate
Sat
Non-sat
mean Vmax
DeepEyes
0.412
0.359
0.053
+0.07
−0.01
81%
13%
37%
−0.02
0.10
+0.05
Mini-o3
0.113
0.051
0.063
+0.09
−0.01
29%
34%
21%
∼0
0.08
+0.19
Qwen3-VL-8B
0.200
0.001
0.198
+0.28
−0.20
56%
29%
64%
∼0
0.55
+0.30
Figure 3: Three-level causal intervention. (a) Policy-level intervention (§3.3.1) toggles the entire T→O→Y subgraph. (b) Trajectory-level intervention (§3.3.2) replaces every real observation with a corrupted one O~i during inference. The policy reacts to corrupted feedback and may produce a different action trace T~1:n~. (c) Step-level intervention (§3.3.3) replaces the real observation Oireal with a counterfactual one Oicf; VEGi=ΔMireal−ΔMicf is a local counterfactual-based estimate of the Natural Indirect Effect along Ti→Oi→Y, capturing the contribution of the returned visual evidence.
Table 4: Per-model group distribution on V∗. Rows sum to 100%. Same (τsat,ϵ)=(0.95,0.01) for every model.
Model
No-call
Mode 1
Calibrated
Mode 2
(CWL)
(LWP)
DeepEyes
19.4
72.8
7.9
0.0
Qwen3-VL-4B
13.1
58.6
24.1
4.2
Qwen3-VL-8B
5.2
70.7
20.9
3.1
Mini-o3
10.5
34.6
45.0
9.9
Figure 4: Per-call scatter of probability gap before call gi−1 vs. VEGi on V∗. Each point is one tool call; color indicates the trajectory’s final correctness (C/I); dashed line marks τsat=0.95.
Table 5: Group-wise decomposition of the policy-level ATE on V∗. Each cell is the contribution fb⋅Δ¯policy,b in pp. The last column is computed from unrounded contributions and matches Table 1; displayed components may not sum exactly due to rounding. (τsat,ϵ)=(0.95,0.01).
Model
No-call
Mode 1
Cal.
Mode 2
ATEpolicy
DeepEyes
0.0
−0.5
+0.5
0.0
0.0
Qwen3-VL-4B
−0.2
+2.0
+3.5
−0.9
+4.4
Qwen3-VL-8B
−0.5
−0.2
+7.6
−0.1
+6.9
Mini-o3
−0.3
+1.9
+3.3
+0.6
+5.5
Figure 5: Within-trajectory dynamics on the V∗ benchmark. (a) Mean VEG by call position k∈{1,2,3,≥4}; Solid curves indicate mean on all calls and dashed curves indicate mean on non-saturated subset only. (b) Per-position Over-Extension Rate (POER, Eq. 5, per-call). Full per-position breakdown in App. C.2.
Table 6: Model checkpoints. HF Repo lists the Hugging Face identifier of the checkpoint used.
Model
Backbone
Tool interface
HF Repo
DeepEyes (31)
Qwen2.5-VL-7B
Interface
ChenShawn/DeepEyes-7B
Pixel Reasoner (23)
Qwen2.5-VL-7B
Interface
TIGER-Lab/PixelReasoner-RL-v1
Mini-o3 (11)
Qwen2.5-VL-7B
Interface
Mini-o3/Mini-o3-7B-v1
Qwen3-VL-8B (2)
Qwen3-VL-8B
Interface
Qwen/Qwen3-VL-8B-Instruct
Qwen3-VL-4B (2)
Qwen3-VL-4B
Interface
Qwen/Qwen3-VL-4B-Instruct
Thyme (28)
Qwen2.5-VL-7B
Code
Kwai-Keye/Thyme-RL
Table 7: Per-model decoding configuration. Values follow each model’s official inference configuration. Both the tool-use and direct rollouts use the same configuration. “Greedy” indicates a deterministic setting. Tmax is the tool-call limit.
Table 9: Trajectory-level corruption-choice ablation on V∗. Each cell reports accuracy under the dynamic intervention, the corresponding Δtraj in pp, and Hit-MaxT.
Model
random-crop
noise
blank
Acc
Δ
Hit
Acc
Δ
Hit
Acc
Δ
Hit
DeepEyes
83.8
+0.4
0%
83.8
+0.4
0%
82.9
-0.4
0%
Pixel Reasoner
82.7
-2.1
0%
82.2
-2.6
0%
83.8
-1.0
0%
Mini-o3
23.6
-64.2
84.8%
20.4
-67.4
78.1%
13.6
-74.2
84.8%
Qwen3-VL-8B
30.4
-60.7
59.2%
28.8
-62.3
64.4%
61.8
-29.3
26.2%
Qwen3-VL-4B
38.2
-48.2
50.3%
48.7
-37.7
38.7%
53.9
-32.5
37.7%
Thyme
82.2
-1.0
0%
82.2
-1.0
0%
81.7
-1.5
0%
Table 10: Mean number of tool calls per item on V∗ under intervention. Contrasts the vanilla baseline with each corruption scheme.
Model
Vanilla
r-crop
noise
blank
DeepEyes
1.0
1.0
1.0
1.0
Pixel Reasoner
0.9
0.9
0.9
0.9
Mini-o3
2.1
9.1
10.1
9.6
Qwen3-VL-8B
1.1
6.5
7.2
4.5
Qwen3-VL-4B
1.1
6.5
5.1
4.9
Thyme
0.1
0.1
0.1
0.1
Table 11: Force-answer Experiment on V∗ under random-crop. Clean is tool-use accuracy; Orig. corr. is the original corrupted accuracy (Table 2); Force-ans. forces an answer at the tool-call limit; Rem. drop is the remaining drop of the force-answer run relative to clean (pp).
Model
Clean
Orig. corr.
Force-ans.
Rem. drop
Mini-o3
87.8
23.6
36.6
51.2
Qwen3-VL-8B
91.1
30.4
73.0
18.1
Qwen3-VL-4B
86.4
38.2
69.6
16.8
Table 12: Per-trajectory peak/trough VEG on V∗ (mean over trajectories). Pooled (A) and split by trajectory correctness (C/I).
mean Vmax
mean Vmin
Model
A
C
I
A
C
I
DeepEyes
+0.05
+0.07
−0.01
+0.05
+0.07
−0.01
Mini-o3
+0.19
+0.21
+0.10
+0.04
+0.08
−0.16
Qwen3-VL-8B
+0.30
+0.37
−0.43
+0.14
+0.21
−0.63
Table 13: Per-position VEG breakdown on V∗. For each model and bin k∈{1,2,3,≥4} we report the number of calls n, mean VEG, fraction of calls with probability gap already past τsat (Sat), the wasted-rate (w), and the harmful-rate (h). Each metric is given (a) over all calls and (b) restricted to non-saturated calls. “–” marks bins with n=0.
All calls
Non-sat (gi−1≤τsat)
Model
k
Sat
n
mean VEG
w / h
n
mean VEG
w / h
DeepEyes
1
36.4%
154
+0.054
81.2% / 5.2%
98
+0.096
73.5% / 7.1%
2
–
0
–
– / –
0
–
– / –
3
–
0
–
– / –
0
–
– / –
≥4
–
0
–
– / –
0
–
– / –
Mini-o3
1
22.8%
171
+0.104
30.4% / 11.1%
132
+0.135
16.7% / 14.4%
2
25.0%
100
+0.070
29.0% / 16.0%
75
+0.098
16.0% / 17.3%
3
23.8%
42
+0.032
31.0% / 19.0%
32
+0.041
18.8% / 21.9%
≥4
12.7%
126
+0.011
27.0% / 20.6%
110
+0.015
21.8% / 21.8%
Qwen3-VL-8B
1
69.1%
181
+0.196
59.1% / 8.3%
56
+0.632
10.7% / 19.6%
2
56.6%
53
+0.273
47.2% / 5.7%
23
+0.569
8.7% / 8.7%
3
46.2%
13
+0.271
46.2% / 0.0%
7
+0.500
28.6% / 0.0%
≥4 †
47.1%
17
+0.033
52.9% / 23.5%
9
+0.062
11.1% / 44.4%
Table 14: Open-ended VEGopen on VisualProbe. Median (IQR) in nats per gold token, over all valid calls and split by trajectory correctness.
Model
All calls
Correct
Incorrect
DeepEyes
+0.001 (0.909)
+0.001 (0.549)
−0.033 (1.098)
Mini-o3
−0.027 (0.813)
+0.012 (0.656)
−0.165 (1.225)
Qwen3-VL-8B
+0.000 (2.424)
+0.017 (1.264)
−0.416 (4.317)
Table 15: Sensitivity to the near-zero VEG threshold ϵ on V∗. τsat=0.95 is fixed. “Cal. contrib” is fCalΔ¯policy,Cal in pp, and ATEpolicy is the row-sum contribution in pp.
Model
ϵ
No-call
Mode 1
Cal.
Mode 2
Cal. contrib
ATEpolicy
DeepEyes
0.005
19.4
72.3
8.4
0.0
+0.5
+0.0
DeepEyes
0.01
19.4
72.8
7.9
0.0
+0.5
+0.0
DeepEyes
0.02
19.4
73.8
6.8
0.0
+0.0
+0.0
DeepEyes
0.05
19.4
73.8
6.8
0.0
+0.0
+0.0
Mini-o3
0.005
10.5
29.8
48.7
11.0
+3.5
+5.5
Mini-o3
0.01
10.5
34.6
45.0
9.9
+3.3
+5.5
Mini-o3
0.02
10.5
38.2
43.5
7.9
+3.7
+5.5
Mini-o3
0.05
10.5
45.0
39.3
5.2
+3.5
+5.5
Qwen3-VL-8B
0.005
5.2
70.7
20.9
3.1
+7.6
+6.9
Qwen3-VL-8B
0.01
5.2
70.7
20.9
3.1
+7.6
+6.9
Qwen3-VL-8B
0.02
5.2
71.7
20.4
2.6
+7.6
+6.9
Qwen3-VL-8B
0.05
5.2
71.7
20.4
2.6
+7.6
+6.9
Qwen3-VL-4B
0.005
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.01
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.02
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.05
13.1
61.3
23.0
2.6
+3.5
+4.4
Table 16: Sensitivity to the saturation threshold τsat on V∗. ϵ=0.01 is fixed. Columns follow Table 15.
Model
τsat
No-call
Mode 1
Cal.
Mode 2
Cal. contrib
ATEpolicy
DeepEyes
0.80
19.4
73.3
7.3
0.0
+1.0
+0.0
DeepEyes
0.90
19.4
72.8
7.9
0.0
+0.5
+0.0
DeepEyes
0.95
19.4
72.8
7.9
0.0
+0.5
+0.0
DeepEyes
0.97
19.4
72.3
8.4
0.0
+0.5
+0.0
DeepEyes
0.99
19.4
71.7
8.9
0.0
+0.5
+0.0
Mini-o3
0.80
10.5
43.5
33.0
13.1
+2.0
+5.5
Mini-o3
0.90
10.5
36.1
41.4
12.0
+3.1
+5.5
Mini-o3
0.95
10.5
34.6
45.0
9.9
+3.3
+5.5
Mini-o3
0.97
10.5
32.5
48.2
8.9
+3.4
+5.5
Mini-o3
0.99
10.5
32.5
52.4
4.7
+3.5
+5.5
Qwen3-VL-8B
0.80
5.2
72.3
18.3
4.2
+7.4
+6.9
Qwen3-VL-8B
0.90
5.2
70.7
19.9
4.2
+7.6
+6.9
Qwen3-VL-8B
0.95
5.2
70.7
20.9
3.1
+7.6
+6.9
Qwen3-VL-8B
0.97
5.2
69.6
21.5
3.7
+7.6
+6.9
Qwen3-VL-8B
0.99
5.2
66.5
24.6
3.7
+7.8
+6.9
Qwen3-VL-4B
0.80
13.1
62.8
20.4
3.7
+3.7
+4.4
Qwen3-VL-4B
0.90
13.1
61.3
22.5
3.1
+3.8
+4.4
Qwen3-VL-4B
0.95
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.97
13.1
56.0
26.2
4.7
+3.5
+4.4
Qwen3-VL-4B
0.99
13.1
54.5
28.8
3.7
+3.5
+4.4
Table 17: Per-model group decomposition of the policy-level ATE on V∗. For each model: fb is the fraction of group b (%); Δ¯policy,b is the mean accuracy gap between πtool and πdirect on the trajectories in group b (pp); contrib =fb⋅Δ¯policy,b (pp). Total is from unrounded contributions and may differ slightly from the sum of displayed cells due to rounding. “–” marks an empty group on this benchmark.
Model
No-call
Mode 1
Cal.
Mode 2
Total
DeepEyes
fb (%)
19.4
72.8
7.9
0.0
100.0
Δ¯policy,b (pp)
0.0
−0.7
+6.7
−
–
contrib (pp)
0.0
−0.5
+0.5
0.0
0.0
Qwen3-VL-4B
fb (%)
13.1
58.6
24.1
4.2
100.0
Δ¯policy,b (pp)
−1.5
+3.3
+14.4
−20.3
–
contrib (pp)
−0.2
+2.0
+3.5
−0.9
+4.4
Qwen3-VL-8B
fb (%)
5.2
70.7
20.9
3.1
100.0
Δ¯policy,b (pp)
−8.8
−0.3
+36.2
−2.1
–
contrib (pp)
−0.5
−0.2
+7.6
−0.1
+6.9
Mini-o3
fb (%)
10.5
34.6
45.0
9.9
100.0
Δ¯policy,b (pp)
−2.5
+5.5
+7.3
+5.9
–
contrib (pp)
−0.3
+1.9
+3.3
+0.6
+5.5
Table 18: Step-level VEG on HR-Bench-4K. VEG C / I splits by trajectory correctness; Sat-rate is the fraction of saturated calls; mean Vmax is the per-trajectory best call averaged over trajectories.
Model
Valid calls
VEG all
VEG C / I
|VEG|<0.01
Sat-rate
mean Vmax
Pattern
DeepEyes
769
−0.011
+0.004 / −0.048
86.9%
34.9%
−0.005
Mode-1-heavy
Mini-o3
2398
+0.030
+0.115 / −0.023
25.4%
12.6%
+0.171
Calibrated-heavy, Mode 2
Qwen3-VL-8B
1049
+0.036
+0.099 / −0.150
64.6%
58.3%
+0.143
Saturation-driven Mode 1
Table 19: Group distribution on HR-Bench-4K. Rows sum to 100%; (τsat,ϵ)=(0.95,0.01).
Model
No-call
Mode 1 (CWL)
Calibrated
Mode 2 (LWP)
DeepEyes
9.5
85.6
4.9
0.0
Mini-o3
16.1
25.8
42.8
15.4
Qwen3-VL-8B
29.8
58.4
10.1
1.6
Table 20: Group-wise ATE decomposition on HR-Bench-4K (pp). Rows sum to the total ATE.
The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.
