AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
arXiv:2608.062702026-08-05
AI models that 'zoom into images' often don't actually use what they see to change their answers
Multimodal AI models that crop and zoom into images while reasoning were tested to see whether the returned image crops causally affect their final answers. Using a three-level intervention (policy, trajectory, step) across six models and five benchmarks, the researchers found that accuracy gains from image tool-use are mostly concentrated in a small 'calibrated' minority of cases, while many others either call the tool without actually using what it shows (Calling Without Looking) or use it but keep calling with no coherent stopping plan (Looking Without Planning). The authors call this the 'illusion of visual tool-use.'
METAL LAB explanatory visual
Three-level causal audit of visual tool-use
Evidence statusMeasured results reported
Policy-level interventionCompares accuracy of tool-using models against direct-inference models to gauge the overall effect of having tools available
Trajectory-level interventionCorrupts every image returned during a run to see how much accuracy drops, revealing the total effect of the observation evidence
Step-level intervention (VEG)Swaps out a single returned image crop for an irrelevant one to isolate how much that one observation shifts answer confidence
Four-group diagnosticSorts each run into No-call, Calling Without Looking, Looking Without Planning, or Calibrated to decompose where accuracy gains come from
Illusion conclusionShows that overall accuracy gains look broad, but the causal benefit is concentrated almost entirely in the small Calibrated group
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
The researchers built a causal graph to separate cases where a returned cropped image genuinely changes the model's answer from cases where merely issuing a tool call (regardless of what it shows) shifts the answer.
They designed three levels of intervention: policy-level (comparing tool-use to direct inference), trajectory-level (corrupting every returned image during a run), and step-level (swapping out just one returned image for a counterfactual crop), introducing a metric called Visual Evidence Gain (VEG) to isolate each single observation's contribution.
They tested six models (DeepEyes, Pixel Reasoner, Mini-o3, Qwen3-VL-4B/8B, Thyme) on five fine-grained perception benchmarks (V*, HR-Bench 4K/8K, VisualProbe, MME-RealWorld-Lite).
At the policy level, DeepEyes showed almost no benefit from tool-use versus direct inference, while Mini-o3 and Qwen3-VL-8B showed large gains, up to +21.3 percentage points on VisualProbe.
Sorting trajectories into four groups (No-call, Calling Without Looking, Looking Without Planning, Calibrated) and decomposing the overall accuracy gain showed that positive gains came almost entirely from the small Calibrated group across models.
Figure 1: The paradox of visual tool-use. (a) Our experiments show that, despite far more tokens, tool-use policies yield marginal or no improvements over direct inference. (b) A qualitative failure on a V∗ instance (by Mini-o3): the direct prediction is correct, but tool-augmented inference repeatedly crops irrelevant regions and leads to no valid answer or a wrong answer.
Table 1: Policy-level intervention: tool-use vs. direct inference. Acc reports accuracy (%) under πtool / πdirect; Δ is the policy-level ATE in percentage points (pp) (Eq. 1). VisualProbe combines Easy/Medium/Hard levels.
Model
V∗
HR-Bench-4K
HR-Bench-8K
VisualProbe
MME-RealWorld-Lite
Interface-based
DeepEyes
Acc
83.3/83.3
75.0/76.4
72.0/72.6
42.9/42.9
54.7/54.9
Δ
0.0
−1.4
−0.6
+0.0
−0.2
Pixel Reasoner
Acc
84.8/81.2
74.0/74.4
68.2/64.0
37.3/29.7
64.4/54.4
Δ
+3.6
−0.4
+4.2
+7.6
+10.0
Mini-o3
Acc
87.8/82.3
76.8/72.0
73.2/67.4
55.0/33.7
65.5/53.6
Δ
+5.5
+4.8
+5.8
+21.3
+11.9
Qwen3-VL-8B
Acc
91.1/84.2
83.9/78.9
79.5/75.7
48.6/36.9
51.0/54.7
Δ
+6.9
+5.0
+3.8
+11.7
−3.7
Qwen3-VL-4B
Acc
86.4/82.0
80.1/80.1
75.3/76.9
45.7/38.9
49.0/50.3
Δ
+4.4
0.0
−1.6
+6.8
−1.3
Code-based
Thyme
Acc
83.2/80.2
78.3/74.7
73.0/68.2
46.2/43.7
55.2/48.9
Δ
+3.0
+3.6
+4.8
+2.6
+6.3
(b) The Zoom-Loop Trap
Table 2: Trajectory-level intervention under random-crop. Acc reports accuracy (%) under πtool / the dynamic intervention do(Oi=O~i∀i); Δ is the trajectory-level ATE on accuracy in pp (Eq. 2). The rightmost column reports Hit-MaxT, the fraction of rollouts that hit the tool-call limit.
Model
V∗
HR-Bench-4K
HR-Bench-8K
VisualProbe
MME-RealWorld-Lite
Hit-MaxT (V∗)
Interface-based
DeepEyes
Acc
83.3/83.8
75.0/74.1
72.0/70.4
42.9/44.2
54.7/54.6
0%
Δ
+0.4
−0.9
−1.6
+1.3
−0.1
Pixel Reasoner
Acc
84.8/82.7
74.0/70.8
68.2/66.1
37.3/33.7
64.4/49.3
0%
Δ
−2.1
−3.2
−2.1
−3.6
−15.1
Mini-o3
Acc
87.8/23.6
76.8/20.4
73.2/15.8
55.0/2.4
65.5/15.1
84.8%
Δ
−64.2
−56.4
−57.4
−52.6
−50.4
Qwen3-VL-8B
Acc
91.1/30.4
83.9/46.4
79.5/41.0
48.6/11.2
51.0/29.7
59.2%
Δ
−60.7
−37.5
−38.5
−37.4
−21.3
Qwen3-VL-4B
Acc
86.4/38.2
80.1/59.3
75.3/46.6
45.7/8.8
49.0/34.3
50.3%
Δ
−48.2
−20.8
−28.7
−36.9
−14.7
Code-based
Thyme
Acc
83.2/82.2
78.3/77.9
73.0/74.1
46.2/45.6
55.2/56.2
0%
Δ
−1.0
−0.4
+1.1
−0.6
+1.0
Figure 2: Causal graph for visual tool-use. (a) The single-step causal graph for a single tool-use step over (I,Q,T,O,Y), highlighting the direct path (I,Q)→Y (blue), the observation-mediated path T→O→Y (green) and the action-induced shortcut T→Y (red). (b) A full trajectory composes n such atomic graphs.
Table 3: Step-level intervention on V∗. Marginal Gain (MG) is reported with the original observation (ΔMreal) and with the counterfactual observation (ΔMcf). Visual Evidence Gain (VEG) isolates the contribution of the visual content; we report it pooled over All calls and split by whether the trajectory is ultimately Correct or Incorrect. Distribution shows the fraction of calls that are near-zero (|VEG|<0.01) or substantial (|VEG|>0.1). By saturation conditions VEG on pre-call confidence: Sat-rate is the fraction of calls whose pre-call probability gap gi−1 already exceeds 0.95, and Sat/Non-sat are mean VEG conditional on (non-)saturation. mean Vmax is each trajectory’s best per-call VEG, averaged across trajectories, capturing how much the most useful call in a trajectory contributes on average.
MG (mean)
VEG all-calls (mean)
Distribution
By saturation
Model
ΔMreal
ΔMcf
A
C
I
|VEG|<0.01
|VEG|>0.1
Sat-rate
Sat
Non-sat
mean Vmax
DeepEyes
0.412
0.359
0.053
+0.07
−0.01
81%
13%
37%
−0.02
0.10
+0.05
Mini-o3
0.113
0.051
0.063
+0.09
−0.01
29%
34%
21%
∼0
0.08
+0.19
Qwen3-VL-8B
0.200
0.001
0.198
+0.28
−0.20
56%
29%
64%
∼0
0.55
+0.30
Figure 3: Three-level causal intervention. (a) Policy-level intervention (§3.3.1) toggles the entire T→O→Y subgraph. (b) Trajectory-level intervention (§3.3.2) replaces every real observation with a corrupted one O~i during inference. The policy reacts to corrupted feedback and may produce a different action trace T~1:n~. (c) Step-level intervention (§3.3.3) replaces the real observation Oireal with a counterfactual one Oicf; VEGi=ΔMireal−ΔMicf is a local counterfactual-based estimate of the Natural Indirect Effect along Ti→Oi→Y, capturing the contribution of the returned visual evidence.
Table 4: Per-model group distribution on V∗. Rows sum to 100%. Same (τsat,ϵ)=(0.95,0.01) for every model.
Model
No-call
Mode 1
Calibrated
Mode 2
(CWL)
(LWP)
DeepEyes
19.4
72.8
7.9
0.0
Qwen3-VL-4B
13.1
58.6
24.1
4.2
Qwen3-VL-8B
5.2
70.7
20.9
3.1
Mini-o3
10.5
34.6
45.0
9.9
Figure 4: Per-call scatter of probability gap before call gi−1 vs. VEGi on V∗. Each point is one tool call; color indicates the trajectory’s final correctness (C/I); dashed line marks τsat=0.95.
Table 5: Group-wise decomposition of the policy-level ATE on V∗. Each cell is the contribution fb⋅Δ¯policy,b in pp. The last column is computed from unrounded contributions and matches Table 1; displayed components may not sum exactly due to rounding. (τsat,ϵ)=(0.95,0.01).
Model
No-call
Mode 1
Cal.
Mode 2
ATEpolicy
DeepEyes
0.0
−0.5
+0.5
0.0
0.0
Qwen3-VL-4B
−0.2
+2.0
+3.5
−0.9
+4.4
Qwen3-VL-8B
−0.5
−0.2
+7.6
−0.1
+6.9
Mini-o3
−0.3
+1.9
+3.3
+0.6
+5.5
Figure 5: Within-trajectory dynamics on the V∗ benchmark. (a) Mean VEG by call position k∈{1,2,3,≥4}; Solid curves indicate mean on all calls and dashed curves indicate mean on non-saturated subset only. (b) Per-position Over-Extension Rate (POER, Eq. 5, per-call). Full per-position breakdown in App. C.2.
Table 6: Model checkpoints. HF Repo lists the Hugging Face identifier of the checkpoint used.
Model
Backbone
Tool interface
HF Repo
DeepEyes (31)
Qwen2.5-VL-7B
Interface
ChenShawn/DeepEyes-7B
Pixel Reasoner (23)
Qwen2.5-VL-7B
Interface
TIGER-Lab/PixelReasoner-RL-v1
Mini-o3 (11)
Qwen2.5-VL-7B
Interface
Mini-o3/Mini-o3-7B-v1
Qwen3-VL-8B (2)
Qwen3-VL-8B
Interface
Qwen/Qwen3-VL-8B-Instruct
Qwen3-VL-4B (2)
Qwen3-VL-4B
Interface
Qwen/Qwen3-VL-4B-Instruct
Thyme (28)
Qwen2.5-VL-7B
Code
Kwai-Keye/Thyme-RL
Table 7: Per-model decoding configuration. Values follow each model’s official inference configuration. Both the tool-use and direct rollouts use the same configuration. “Greedy” indicates a deterministic setting. Tmax is the tool-call limit.
Table 9: Trajectory-level corruption-choice ablation on V∗. Each cell reports accuracy under the dynamic intervention, the corresponding Δtraj in pp, and Hit-MaxT.
Model
random-crop
noise
blank
Acc
Δ
Hit
Acc
Δ
Hit
Acc
Δ
Hit
DeepEyes
83.8
+0.4
0%
83.8
+0.4
0%
82.9
-0.4
0%
Pixel Reasoner
82.7
-2.1
0%
82.2
-2.6
0%
83.8
-1.0
0%
Mini-o3
23.6
-64.2
84.8%
20.4
-67.4
78.1%
13.6
-74.2
84.8%
Qwen3-VL-8B
30.4
-60.7
59.2%
28.8
-62.3
64.4%
61.8
-29.3
26.2%
Qwen3-VL-4B
38.2
-48.2
50.3%
48.7
-37.7
38.7%
53.9
-32.5
37.7%
Thyme
82.2
-1.0
0%
82.2
-1.0
0%
81.7
-1.5
0%
Table 10: Mean number of tool calls per item on V∗ under intervention. Contrasts the vanilla baseline with each corruption scheme.
Model
Vanilla
r-crop
noise
blank
DeepEyes
1.0
1.0
1.0
1.0
Pixel Reasoner
0.9
0.9
0.9
0.9
Mini-o3
2.1
9.1
10.1
9.6
Qwen3-VL-8B
1.1
6.5
7.2
4.5
Qwen3-VL-4B
1.1
6.5
5.1
4.9
Thyme
0.1
0.1
0.1
0.1
Table 11: Force-answer Experiment on V∗ under random-crop. Clean is tool-use accuracy; Orig. corr. is the original corrupted accuracy (Table 2); Force-ans. forces an answer at the tool-call limit; Rem. drop is the remaining drop of the force-answer run relative to clean (pp).
Model
Clean
Orig. corr.
Force-ans.
Rem. drop
Mini-o3
87.8
23.6
36.6
51.2
Qwen3-VL-8B
91.1
30.4
73.0
18.1
Qwen3-VL-4B
86.4
38.2
69.6
16.8
Table 12: Per-trajectory peak/trough VEG on V∗ (mean over trajectories). Pooled (A) and split by trajectory correctness (C/I).
mean Vmax
mean Vmin
Model
A
C
I
A
C
I
DeepEyes
+0.05
+0.07
−0.01
+0.05
+0.07
−0.01
Mini-o3
+0.19
+0.21
+0.10
+0.04
+0.08
−0.16
Qwen3-VL-8B
+0.30
+0.37
−0.43
+0.14
+0.21
−0.63
Table 13: Per-position VEG breakdown on V∗. For each model and bin k∈{1,2,3,≥4} we report the number of calls n, mean VEG, fraction of calls with probability gap already past τsat (Sat), the wasted-rate (w), and the harmful-rate (h). Each metric is given (a) over all calls and (b) restricted to non-saturated calls. “–” marks bins with n=0.
All calls
Non-sat (gi−1≤τsat)
Model
k
Sat
n
mean VEG
w / h
n
mean VEG
w / h
DeepEyes
1
36.4%
154
+0.054
81.2% / 5.2%
98
+0.096
73.5% / 7.1%
2
–
0
–
– / –
0
–
– / –
3
–
0
–
– / –
0
–
– / –
≥4
–
0
–
– / –
0
–
– / –
Mini-o3
1
22.8%
171
+0.104
30.4% / 11.1%
132
+0.135
16.7% / 14.4%
2
25.0%
100
+0.070
29.0% / 16.0%
75
+0.098
16.0% / 17.3%
3
23.8%
42
+0.032
31.0% / 19.0%
32
+0.041
18.8% / 21.9%
≥4
12.7%
126
+0.011
27.0% / 20.6%
110
+0.015
21.8% / 21.8%
Qwen3-VL-8B
1
69.1%
181
+0.196
59.1% / 8.3%
56
+0.632
10.7% / 19.6%
2
56.6%
53
+0.273
47.2% / 5.7%
23
+0.569
8.7% / 8.7%
3
46.2%
13
+0.271
46.2% / 0.0%
7
+0.500
28.6% / 0.0%
≥4 †
47.1%
17
+0.033
52.9% / 23.5%
9
+0.062
11.1% / 44.4%
Table 14: Open-ended VEGopen on VisualProbe. Median (IQR) in nats per gold token, over all valid calls and split by trajectory correctness.
Model
All calls
Correct
Incorrect
DeepEyes
+0.001 (0.909)
+0.001 (0.549)
−0.033 (1.098)
Mini-o3
−0.027 (0.813)
+0.012 (0.656)
−0.165 (1.225)
Qwen3-VL-8B
+0.000 (2.424)
+0.017 (1.264)
−0.416 (4.317)
Table 15: Sensitivity to the near-zero VEG threshold ϵ on V∗. τsat=0.95 is fixed. “Cal. contrib” is fCalΔ¯policy,Cal in pp, and ATEpolicy is the row-sum contribution in pp.
Model
ϵ
No-call
Mode 1
Cal.
Mode 2
Cal. contrib
ATEpolicy
DeepEyes
0.005
19.4
72.3
8.4
0.0
+0.5
+0.0
DeepEyes
0.01
19.4
72.8
7.9
0.0
+0.5
+0.0
DeepEyes
0.02
19.4
73.8
6.8
0.0
+0.0
+0.0
DeepEyes
0.05
19.4
73.8
6.8
0.0
+0.0
+0.0
Mini-o3
0.005
10.5
29.8
48.7
11.0
+3.5
+5.5
Mini-o3
0.01
10.5
34.6
45.0
9.9
+3.3
+5.5
Mini-o3
0.02
10.5
38.2
43.5
7.9
+3.7
+5.5
Mini-o3
0.05
10.5
45.0
39.3
5.2
+3.5
+5.5
Qwen3-VL-8B
0.005
5.2
70.7
20.9
3.1
+7.6
+6.9
Qwen3-VL-8B
0.01
5.2
70.7
20.9
3.1
+7.6
+6.9
Qwen3-VL-8B
0.02
5.2
71.7
20.4
2.6
+7.6
+6.9
Qwen3-VL-8B
0.05
5.2
71.7
20.4
2.6
+7.6
+6.9
Qwen3-VL-4B
0.005
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.01
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.02
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.05
13.1
61.3
23.0
2.6
+3.5
+4.4
Table 16: Sensitivity to the saturation threshold τsat on V∗. ϵ=0.01 is fixed. Columns follow Table 15.
Model
τsat
No-call
Mode 1
Cal.
Mode 2
Cal. contrib
ATEpolicy
DeepEyes
0.80
19.4
73.3
7.3
0.0
+1.0
+0.0
DeepEyes
0.90
19.4
72.8
7.9
0.0
+0.5
+0.0
DeepEyes
0.95
19.4
72.8
7.9
0.0
+0.5
+0.0
DeepEyes
0.97
19.4
72.3
8.4
0.0
+0.5
+0.0
DeepEyes
0.99
19.4
71.7
8.9
0.0
+0.5
+0.0
Mini-o3
0.80
10.5
43.5
33.0
13.1
+2.0
+5.5
Mini-o3
0.90
10.5
36.1
41.4
12.0
+3.1
+5.5
Mini-o3
0.95
10.5
34.6
45.0
9.9
+3.3
+5.5
Mini-o3
0.97
10.5
32.5
48.2
8.9
+3.4
+5.5
Mini-o3
0.99
10.5
32.5
52.4
4.7
+3.5
+5.5
Qwen3-VL-8B
0.80
5.2
72.3
18.3
4.2
+7.4
+6.9
Qwen3-VL-8B
0.90
5.2
70.7
19.9
4.2
+7.6
+6.9
Qwen3-VL-8B
0.95
5.2
70.7
20.9
3.1
+7.6
+6.9
Qwen3-VL-8B
0.97
5.2
69.6
21.5
3.7
+7.6
+6.9
Qwen3-VL-8B
0.99
5.2
66.5
24.6
3.7
+7.8
+6.9
Qwen3-VL-4B
0.80
13.1
62.8
20.4
3.7
+3.7
+4.4
Qwen3-VL-4B
0.90
13.1
61.3
22.5
3.1
+3.8
+4.4
Qwen3-VL-4B
0.95
13.1
58.6
24.1
4.2
+3.5
+4.4
Qwen3-VL-4B
0.97
13.1
56.0
26.2
4.7
+3.5
+4.4
Qwen3-VL-4B
0.99
13.1
54.5
28.8
3.7
+3.5
+4.4
Table 17: Per-model group decomposition of the policy-level ATE on V∗. For each model: fb is the fraction of group b (%); Δ¯policy,b is the mean accuracy gap between πtool and πdirect on the trajectories in group b (pp); contrib =fb⋅Δ¯policy,b (pp). Total is from unrounded contributions and may differ slightly from the sum of displayed cells due to rounding. “–” marks an empty group on this benchmark.
Model
No-call
Mode 1
Cal.
Mode 2
Total
DeepEyes
fb (%)
19.4
72.8
7.9
0.0
100.0
Δ¯policy,b (pp)
0.0
−0.7
+6.7
−
–
contrib (pp)
0.0
−0.5
+0.5
0.0
0.0
Qwen3-VL-4B
fb (%)
13.1
58.6
24.1
4.2
100.0
Δ¯policy,b (pp)
−1.5
+3.3
+14.4
−20.3
–
contrib (pp)
−0.2
+2.0
+3.5
−0.9
+4.4
Qwen3-VL-8B
fb (%)
5.2
70.7
20.9
3.1
100.0
Δ¯policy,b (pp)
−8.8
−0.3
+36.2
−2.1
–
contrib (pp)
−0.5
−0.2
+7.6
−0.1
+6.9
Mini-o3
fb (%)
10.5
34.6
45.0
9.9
100.0
Δ¯policy,b (pp)
−2.5
+5.5
+7.3
+5.9
–
contrib (pp)
−0.3
+1.9
+3.3
+0.6
+5.5
Table 18: Step-level VEG on HR-Bench-4K. VEG C / I splits by trajectory correctness; Sat-rate is the fraction of saturated calls; mean Vmax is the per-trajectory best call averaged over trajectories.
Model
Valid calls
VEG all
VEG C / I
|VEG|<0.01
Sat-rate
mean Vmax
Pattern
DeepEyes
769
−0.011
+0.004 / −0.048
86.9%
34.9%
−0.005
Mode-1-heavy
Mini-o3
2398
+0.030
+0.115 / −0.023
25.4%
12.6%
+0.171
Calibrated-heavy, Mode 2
Qwen3-VL-8B
1049
+0.036
+0.099 / −0.150
64.6%
58.3%
+0.143
Saturation-driven Mode 1
Table 19: Group distribution on HR-Bench-4K. Rows sum to 100%; (τsat,ϵ)=(0.95,0.01).
Model
No-call
Mode 1 (CWL)
Calibrated
Mode 2 (LWP)
DeepEyes
9.5
85.6
4.9
0.0
Mini-o3
16.1
25.8
42.8
15.4
Qwen3-VL-8B
29.8
58.4
10.1
1.6
Table 20: Group-wise ATE decomposition on HR-Bench-4K (pp). Rows sum to the total ATE.
Model
No-call
Mode 1
Cal.
Mode 2
Total
DeepEyes
−0.5
−1.5
+0.6
0.0
−1.4
Mini-o3
+0.5
+0.5
+3.4
+0.4
+4.8
Qwen3-VL-8B
+1.4
−0.9
+4.4
+0.1
+5.0
Findings
Policy-level comparisons of tool-use versus direct inference varied sharply by model: essentially null for DeepEyes, and up to +21.3 percentage points for Mini-o3 and Qwen3-VL-8B on VisualProbe.
Corrupting every returned image during a run (trajectory-level intervention) barely changed accuracy for DeepEyes and Thyme, but caused large accuracy drops for Mini-o3 and both Qwen3-VL models, partly because they hit the tool-call limit more often without a valid answer.
Step-level VEG measurements showed DeepEyes' visual evidence contribution was near zero across the board (structurally inactive), Mini-o3 had a positive best-case contribution per trajectory but a low average (informative but diluted), and Qwen3-VL-8B showed near-zero contribution once confidence was already saturated but large contribution otherwise.
Decomposing the policy-level accuracy gain on V* by trajectory group showed that the Calibrated group was the only one consistently contributing positively across models, accounting for most of each model's overall gain.
The same qualitative pattern (Calibrated group driving the gain) was reproduced on HR-Bench-4K, indicating it is not specific to the V* benchmark.
Where it can be used
Evaluating multimodal AI systems that use visual tools by checking whether tool calls causally affect answers, not just whether overall accuracy improved.
Designing inference-time rules that skip tool calls for already-confident cases or enforce early stopping for trajectories that keep calling without a coherent plan.
Informing reward design in reinforcement learning for tool-use, by incorporating step-level evidence-of-use signals like VEG instead of relying only on outcome-based rewards.
Limits and open work
All experiments used open-source models; step-level VEG requires access to internal token probabilities, so it could not be tested on closed-source models like OpenAI o3/o4-mini.
The analysis focused only on the crop-and-zoom tool; whether the same failure patterns hold for other tools like OCR, object detection, or video frame selection was not verified.
The proposed explanation that outcome-only reinforcement learning causes these failure modes is a hypothesis consistent with the observations, not something established through controlled training experiments that vary only the reward signal.
VEG measurements for open-ended (non-multiple-choice) questions use a different, unbounded scale, so absolute values there are not directly comparable, and the same saturation thresholds cannot simply be reused.
Why it matters
This matters because benchmark scores showing that AI 'looks closer' at images can create a misleading impression that the tool is being used effectively everywhere, when in fact most tool calls may not causally matter. Anyone building or evaluating multimodal AI systems with visual tools should check whether the tool's output actually drives the answer, not just whether accuracy went up.
Terms in this paper
thinking-with-images · A paradigm where an AI actively manipulates images (e.g., cropping and zooming) as part of its reasoning process
causal graph · A diagram showing which variables actually cause changes in others, used to separate real cause-effect from mere correlation
Visual Evidence Gain (VEG) · A metric measuring how much a single returned image crop actually shifts the model's confidence toward the correct answer
Calling Without Looking / Looking Without Planning · Two failure modes: calling a tool without the returned image mattering to the answer, versus using informative images but with no coherent plan for when to stop calling
policy miscalibration · A mismatch between when/how an AI decides to call a tool and how genuinely useful that tool's output actually is
Original abstract (English)
The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.
