When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

arXiv:2608.192082026-08-21

Slipping an irrelevant sentence into a prompt shifts multimodal AI answers in a predictable, formula-like way

Researchers tested what happens when a sentence completely unrelated to an image and question is inserted into prompts for multimodal large language models (MLLMs). They found that the gap between 'yes' and 'no' answer probabilities doesn't scatter randomly but instead shifts following a consistent straight-line (affine) formula. Using this pattern, they built a simple after-the-fact correction that can partly undo the distortion.

What they did

The team ran experiments on yes/no visual benchmarks (POPE, AMBER, MME, GQA) using LLaVA-1.5-7B, the Qwen2-VL family (2B and 7B), and InternVL3-8B, inserting unrelated WikiText sentences after the question.
They defined a 'decision margin' as the log-probability gap between the 'Yes' and 'No' answers, and found that the margin after adding irrelevant text (mc) closely follows mc≈a·m0+b relative to the original margin (m0), fitting with high R² across models and datasets.
Framing the irrelevant text as 'possibly related to the image' sharply lowered the slope (a), meaning the model's original visual judgment was more distorted, while the offset (b) stayed consistently negative, pushing answers toward 'No.'
Samples whose original margin was close to zero (i.e., borderline cases) flipped their answer far more often than confident ones; reversing the fitted affine formula largely restored accuracy and F1 on held-out data and cut the flip rate substantially.
In open-ended image description tasks, adding irrelevant text made models describe only the most salient objects while dropping mentions of peripheral objects, producing shorter and more conservative captions.

Figure 1: Overview of the controlled intervention and its behavioral effects. (a) For each image-question pair, we compare a context-free input with a paired context-conditioned input, where an irrelevant context is inserted while the prompt structure is kept unchanged. (b) Across multiple benchmarks, irrelevant context not only reduces predictive accuracy, but also shifts model decisions toward negative responses.

Table 1: Behavioral effects of image-irrelevant context across datasets and models. Irrelevant context generally degrades performance, induces a negative-response shift, and has stronger effects when presented as potentially image-related. Free indicates the context-free condition, while ΔNeu. and ΔCon. report changes under the context-neutral and context-condition setting relative to context-free condition. Yes Rate reports the proportion of positive responses among all answers, indicating the model’s tendency toward affirmative predictions. Flip Rate reports the percentage of samples whose prediction changes. All values are reported in percentage points.
Dataset	Model	Accuracy	Recall	Yes Rate	Flip Rate
Free	ΔNeu.	ΔCon.	Free	ΔNeu.	ΔCon.	Free	ΔNeu.	ΔCon.	Neu.	Con.
MME	LLaVA-1.5-7B	78.5	↑0.1	\cellcolorgray!10↓0.9	80.5	↓7.9	\cellcolorgray!10↓8.4	51.9	↓7.9	\cellcolorgray!10↓7.4	9.0	\cellcolorgray!109.8
Qwen2-VL-2B	79.9	↓10.6	\cellcolorgray!10↓20.8	80.5	↓24.6	\cellcolorgray!10↓40.8	48.7	↓16.8	\cellcolorgray!10↓26.4	20.5	\cellcolorgray!1033.4
Qwen2-VL-7B	87.6	↑0.2	\cellcolorgray!10↓27.5	90.6	↓2.6	\cellcolorgray!10↓69.5	52.7	↓2.8	\cellcolorgray!10↓41.7	4.3	\cellcolorgray!1042.0
InternVL3	87.9	↓3.1	\cellcolorgray!10↓3.7	85.3	↓9.9	\cellcolorgray!10↓10.5	47.3	↓7.0	\cellcolorgray!10↓7.0	7.8	\cellcolorgray!107.9
GQA	LLaVA-1.5-7B	76.4	→0.0	\cellcolorgray!10↓1.9	76.4	↑2.2	\cellcolorgray!10↑7.0	50.7	↑2.3	\cellcolorgray!10↑9.1	7.6	\cellcolorgray!1014.3
Qwen2-VL-2B	79.2	↓6.3	\cellcolorgray!10↓6.2	76.1	↓20.1	\cellcolorgray!10↓18.4	47.6	↓14.3	\cellcolorgray!10↓12.6	16.0	\cellcolorgray!1015.6
Qwen2-VL-7B	82.2	↓1.4	\cellcolorgray!10↓19.8	78.7	↓2.8	\cellcolorgray!10↓48.3	47.4	↓1.6	\cellcolorgray!10↓29.9	3.9	\cellcolorgray!1030.0
InternVL3	81.9	↓1.5	\cellcolorgray!10↓2.0	85.1	↓4.3	\cellcolorgray!10↓6.1	54.2	↓2.9	\cellcolorgray!10↓4.2	8.0	\cellcolorgray!109.4
AMBER	LLaVA-1.5-7B	81.2	↓0.3	\cellcolorgray!10↓1.3	64.8	↓2.4	\cellcolorgray!10↓2.3	28.8	↓1.3	\cellcolorgray!10↓2.9	6.3	\cellcolorgray!105.7
Qwen2-VL-2B	85.3	↓0.7	\cellcolorgray!10↓4.9	78.5	↓9.8	\cellcolorgray!10↓21.6	33.9	↓5.9	\cellcolorgray!10↓10.8	7.2	\cellcolorgray!1012.3
Qwen2-VL-7B	86.9	↑0.5	\cellcolorgray!10↓9.4	81.2	↓2.0	\cellcolorgray!10↓43.1	34.1	↓1.8	\cellcolorgray!10↓19.7	2.9	\cellcolorgray!1019.7
InternVL3	88.3	↑0.5	\cellcolorgray!10↑0.1	85.7	↓4.7	\cellcolorgray!10↓6.0	35.8	↓3.7	\cellcolorgray!10↓4.2	5.6	\cellcolorgray!106.0
POPE	LLaVA-1.5-7B	85.5	↓0.7	\cellcolorgray!10↓0.3	76.6	↓2.4	\cellcolorgray!10↓0.9	41.4	↓1.8	\cellcolorgray!10↓0.6	2.3	\cellcolorgray!102.3
Qwen2-VL-2B	87.8	↓1.8	\cellcolorgray!10↓2.0	79.6	↓4.5	\cellcolorgray!10↓5.1	42.1	↓2.8	\cellcolorgray!10↓3.2	3.3	\cellcolorgray!103.6
Qwen2-VL-7B	88.3	↓1.3	\cellcolorgray!10↓8.4	81.1	↓3.4	\cellcolorgray!10↓19.8	43.1	↓2.1	\cellcolorgray!10↓11.6	2.6	\cellcolorgray!1011.6
InternVL3	90.9	↓0.8	\cellcolorgray!10↓0.3	90.7	↓5.3	\cellcolorgray!10↓3.9	50.2	↓4.5	\cellcolorgray!10↓3.6	4.8	\cellcolorgray!104.1

Figure 2: Illustration of the margin-level analysis. (a) The decision margin is defined as the log-probability difference between the positive and negative answers. (b) Adding irrelevant context transforms the context-free margin m0 into the context-conditioned margin mc, which follows an approximate affine relation mc≈am0+b. (c) The affine transformation shifts the effective decision boundary from m0=0 to m0=−b/a. This shift expands the Negative Region and compresses the Positive Region. As a result, samples in the Flip Region, i.e., 0<m0<−b/a, are likely to change from positive to negative predictions, inducing a systematic shift in decision preference.

Table 2: Fitted affine parameters across models and benchmarks. The affine relationship exhibits consistently high goodness of fit, indicating that such affine is structural and stable.
Model	Dataset	a	b	R2	r
LLaVA-1.5-7B	POPE	0.55	-1.05	0.90	0.95
AMBER	0.52	-1.15	0.79	0.89
MME	0.54	-1.08	0.83	0.91
GQA	0.48	-0.92	0.74	0.86
Qwen2-VL-2B	POPE	0.56	-1.58	0.86	0.93
AMBER	0.59	-1.61	0.86	0.93
MME	0.56	-1.13	0.74	0.86
GQA	0.62	-1.55	0.78	0.88

Figure 3: Visualization of three input conditions. These three conditions represent different degrees to which the irrelevant context is framed as potentially image-related.

Table 3: In-domain post-hoc affine calibration under WikiText-after-possibly context. Affine parameters are fitted on calibration splits and evaluated on held-out test splits. We report the Accuracy and Flip rate with percentage.
Dataset	Origin	Con.	Cal.	Con. Flip	Cal. Flip
POPE	86.1	78.0	84.6	16.6	6.1
AMBER	82.7	74.9	79.7	29.2	13.6
MME	78.7	68.5	76.5	30.2	12.6
GQA	73.2	69.2	72.3	36.3	14.8

Figure 4: Affine margin patterns for LLaVA-1.5-7B across four benchmarks. Each panel plots the context-free margin mo against the context-conditioned margin mc. Points are colored by prediction transitions between the two conditions, and the solid line shows the fitted affine relation mc=amo+b; the dashed line denotes the identity relation mc=mo. The consistently high R2 and correlation values indicate that textual context induces a structured affine transformation of the original decision margin rather than unstructured perturbations. Notably, changing the semantic framing from Neutral to Condition substantially decreases the fitted slope a, while leaving the negative offset b largely stable. Results are shown for LLaVA.

Table 4: Cross-dataset affine calibration transfer result. Affine parameters are fitted on POPE and directly applied to other benchmarks without target calibration samples. We report the Accuracy and Flip rate with percentage.
Dataset	Origin	Con.	Cal.	Con. Flip	Cal. Flip
AMBER	82.7	75.0	80.1	29.1	14.7
MME	78.7	68.5	76.6	30.4	12.8
GQA	73.2	69.3	71.7	36.2	13.1

Figure 6: Experiment results for Validating the Parameter Interpretation. Points are colored by ground-truth label. Arrows show the mean label-conditioned margin shift from the identity line to the observed context-conditioned mean. The monotonic trend asup>airr>aconf is consistent with the interpretation of a as commitment preservation.

Table 5: Absolute behavioral metrics across tested models and benchmarks. Free denotes context-free input, Neutral denotes WikiText-after-neutral, and Condition denotes WikiText-after-possibly. Part (b) reports the original-style MME score.
Model	Dataset	Free	Neutral	Condition
Acc	F1	Rec.	Prec.	Acc	F1	Rec.	Prec.	Acc	F1	Rec.	Prec.
LLaVA-1.5-7B	POPE	85.5	84.2	76.6	93.5	84.8	83.2	74.2	94.6	85.2	83.8	75.7	93.8
LLaVA-1.5-7B	AMBER	81.2	69.8	64.8	75.8	80.9	68.7	62.4	76.5	79.9	69.2	67.1	71.4
LLaVA-1.5-7B	MME	78.5	78.9	80.5	77.4	78.6	77.2	72.6	82.5	77.6	76.3	72.1	81.0
LLaVA-1.5-7B	GQA-Binary	76.4	76.9	76.4	77.5	76.4	77.4	78.6	76.3	74.5	77.1	83.4	71.7
Qwen2-VL-2B	POPE	87.8	86.8	79.6	95.5	86.0	84.5	75.1	96.4	85.8	84.1	74.5	96.7
Qwen2-VL-2B	AMBER	85.3	78.3	78.5	78.1	84.6	75.1	68.7	82.7	80.4	66.2	56.9	79.1
Qwen2-VL-2B	MME	79.9	80.1	80.5	79.6	69.3	64.6	55.9	76.4	59.1	49.3	39.7	65.0
Qwen2-VL-2B	GQA-Binary	79.2	79.0	76.1	82.1	72.9	68.0	56.0	86.6	73.0	68.7	57.7	84.9
Qwen2-VL-7B	POPE	88.3	87.5	81.1	95.0	87.0	85.8	77.7	95.7	79.9	75.5	61.3	98.4
Qwen2-VL-7B	AMBER	86.9	80.7	81.2	80.2	87.4	80.9	79.2	82.7	77.5	53.3	38.1	88.8
Qwen2-VL-7B	MME	87.6	88.0	90.6	85.5	87.8	87.8	88.0	87.7	60.1	34.6	21.1	95.4
Qwen2-VL-7B	GQA-Binary	82.2	82.0	78.7	85.5	80.8	80.3	75.9	85.2	62.4	45.4	30.4	89.6
InternVL3	POPE	90.9	91.0	90.7	91.3	90.1	89.7	85.4	94.4	90.6	90.3	86.8	94.1
InternVL3	AMBER	88.3	83.1	85.7	80.6	88.8	82.9	81.0	84.9	88.4	82.2	79.7	84.9
InternVL3	MME	87.9	87.6	85.3	90.0	84.8	83.2	75.4	92.7	84.2	82.6	74.8	92.1
InternVL3	GQA-Binary	81.9	82.9	85.1	80.8	80.4	80.9	80.8	81.0	79.9	80.2	79.0	81.4

Figure 7: The affine transformation results across all models and benchmarks.

Table 6: In-domain post-hoc affine calibration under WikiText-after-possibly context. Affine parameters are fitted on calibration splits and evaluated on held-out test splits. Values are percentages averaged over five splits. Flip is relative to Origin predictions.
Model	Dataset	Origin Acc.	Context Acc.	Cal. Acc.	Origin F1	Context F1	Cal. F1	Context Flip	Cal. Flip
LLaVA-1.5-7B	POPE	86.1	78.0	84.6	85.6	72.6	83.2	16.6	6.1
LLaVA-1.5-7B	AMBER	82.7	74.9	79.7	77.5	47.3	71.0	29.2	13.6
LLaVA-1.5-7B	MME	78.7	68.5	76.5	79.2	56.5	76.4	30.2	12.6
LLaVA-1.5-7B	GQA-Binary	73.2	69.2	72.3	77.7	63.5	75.8	36.3	14.8

Figure 8: Residual diagnostics for the affine fit under the WikiText-after-possibly context. Columns denote benchmarks and rows denote models. Each point is a sample residual mc−(am0+b); the red curve is the binned mean residual. All panels use shared symmetric axes so that residual dispersion is directly comparable.

Table 7: Cross-dataset affine calibration transfer on LLaVA-1.5-7B under WikiText-after-possibly context. Source-fitted parameters are applied directly to the target benchmark without target calibration samples. Values are percentages.
Source	Target	Origin Acc.	Cal. Acc.	Origin F1	Cal. F1	Origin Flip	Cal. Flip
POPE	AMBER	75.0	80.1	47.7	70.3	29.1	14.7
POPE	MME	68.5	76.6	56.4	76.2	30.4	12.8
POPE	GQA-Binary	69.3	71.7	63.6	75.9	36.2	13.1
AMBER	POPE	78.2	85.2	72.8	84.0	16.5	5.2
AMBER	MME	68.5	76.5	56.4	77.0	30.4	12.4
AMBER	GQA-Binary	69.3	70.3	63.6	75.6	36.2	12.0
MME	POPE	78.2	84.9	72.8	83.6	16.5	5.6
MME	AMBER	75.0	80.2	47.7	70.9	29.1	14.1
MME	GQA-Binary	69.3	71.0	63.6	75.7	36.2	12.4
GQA-Binary	POPE	78.2	84.3	72.8	82.5	16.5	7.2
GQA-Binary	AMBER	75.0	79.9	47.7	68.4	29.1	16.4
GQA-Binary	MME	68.5	76.2	56.4	74.8	30.4	14.0

Figure 9: A case of our open-ended generation experiment. We use the green mask to indicate the region covered by the model’s prediction under the context-condition setting. After adding irrelevant context, the model tends to describe only salient objects in the image, while peripheral objects are no longer mentioned.

Table 8: Prompt, template, length, and source ablations on POPE with LLaVA-1.5-7B. Acc, Recall, YesRate, and Flip are reported in percentages.
Group	Variant	a	b	R2	Acc	Recall	Flip
Position/template	Wiki before + possibly	0.726	-0.041	0.943	86.4	82.8	3.7
	Wiki after + possibly	0.554	-1.046	0.896	78.2	57.8	16.5
	Wiki before + neutral	0.755	0.084	0.949	86.2	82.7	3.7
	Wiki after + neutral	0.706	-1.243	0.921	80.1	61.6	14.5
Template only	After + possibly	0.583	-0.910	0.944	80.8	63.3	13.5
	After + neutral	0.819	-0.455	0.976	85.1	74.3	6.7
Length	Short Wiki	0.699	-0.000	0.940	86.3	82.4	4.1
	Medium Wiki	0.726	-0.041	0.943	86.4	82.8	3.7
	Long Wiki	0.743	-0.150	0.934	86.3	82.2	4.2
Source/shuffle	Shuffled Wiki	0.796	0.037	0.965	86.7	82.8	3.2
	COCO caption before	0.617	0.192	0.896	85.7	84.7	6.0
	COCO caption after	0.475	-1.204	0.892	74.2	49.7	20.7

Table 9: Label-verbalization controls on POPE with LLaVA-1.5-7B. For each verbalizer pair, margins score the semantically affirmative token against the negative token. Behavioral metrics and Flip are percentages; affine quantities are fitted between Origin and each context condition.
Verbalizer	Condition	Acc	F1	YesRate	Flip	a	b	R2	r
Yes/No	Origin	86.4	86.1	47.1	–	–	–	–	–
Neutral	73.5	64.7	24.7	22.4	0.481	-1.229	0.854	0.924
Possibly	75.3	68.0	26.8	20.4	0.384	-0.871	0.836	0.914
True/False	Origin	85.6	84.7	44.1	–	–	–	–	–
Neutral	80.9	82.5	58.7	16.5	0.600	0.587	0.791	0.889
Possibly	62.5	72.7	86.6	42.6	0.510	1.124	0.747	0.864

Table 10: Natural verbalizer control on AMBER discriminative with LLaVA-1.5-7B using True/False. Behavioral metrics and Flip are reported in percentages.
Condition	Acc	Yes	Flip	a	b	R2
Origin	77.4	26.9	–	–	–	–
Neutral	68.8	58.7	32.9	0.58	0.61	0.66
Possibly	48.4	84.2	57.4	0.47	1.04	0.57

Table 11: Context-role intervention on POPE with LLaVA-1.5-7B. Δm+ and Δm− denote the mean shift mc−m0 for ground-truth Yes and No samples, respectively.
Context	a	b	R2	Δm+	Δm−
Irrelevant	0.554	-1.046	0.896	-1.96	0.16
Supporting	0.626	-0.287	0.873	-0.94	0.61
Conflicting	0.367	-1.228	0.869	-2.63	0.59

Table 12: Context-only prior subtraction on POPE with LLaVA-1.5-7B. We report the affine fit between m0 and the prior-subtracted margin msub=mc−mprior. Acc, F1, YesRate, and Flip are reported in percentages.
Context	Prior estimator	a	b	R2	r	Acc	F1	YesRate	Flip
neutral	context-only blank	0.705	-0.872	0.898	0.948	81.7	78.5	34.5	11.4
neutral	question+context blank	0.700	1.894	0.904	0.951	79.7	82.4	64.9	19.1
neutral	context-only same image	0.705	-0.653	0.902	0.950	82.7	80.0	35.9	10.0
possibly	context-only blank	0.552	-0.233	0.868	0.932	83.3	81.3	38.5	8.2
possibly	question+context blank	0.549	1.599	0.872	0.934	75.5	79.7	70.0	24.3
possibly	context-only same image	0.551	0.103	0.862	0.928	84.8	83.5	41.9	6.4

Table 13: Nested margin-model comparison on POPE with LLaVA-1.5-7B. The context variant is WikiText-after-neutral. Fit metrics evaluate mc prediction on held-out samples. Calibration metrics evaluate the corrected margin as a binary prediction.
Model	R2	MAE	Flip F1	Calib Acc	Calib F1	Calib Flip
Identity	0.522	1.337	0.000	0.800	0.756	0.145
Shift-only	0.760	0.934	0.711	0.854	0.841	0.054
Scale-only	0.642	1.254	0.000	0.800	0.756	0.145
Full affine	0.920	0.518	0.849	0.855	0.842	0.050

Table 14: Local margin analysis under WikiText-after-possibly. For each model and dataset, we report the global affine fit and the flip rate within low-, mid-, and high-margin regions. Flip rates are percentages.
Model	Dataset	Global a	Global R2	Trim R2	Low Flip	Mid Flip	High Flip
LLaVA	POPE	0.554	0.896	0.899	42.2	6.4	0.1
LLaVA	AMBER	0.525	0.786	0.785	52.4	31.6	2.6
LLaVA	MME	0.542	0.830	0.805	52.1	33.0	5.3
LLaVA	GQA	0.476	0.743	0.691	56.3	42.3	9.6
Qwen2-2B	POPE	0.557	0.859	0.841	42.6	15.4	3.1
Qwen2-2B	AMBER	0.592	0.863	0.848	46.0	18.7	3.0
Qwen2-2B	MME	0.561	0.740	0.730	42.8	21.7	5.9
Qwen2-2B	GQA	0.615	0.775	0.750	53.6	33.0	5.5

Table 15: Random-context-seed robustness on POPE with LLaVA-1.5-7B under the WikiText-after-possibly condition. Behavioral metrics and Flip are reported in percentages.
Seed	a	b	R2	r
13	0.5541	-1.0441	0.8968	0.9470
21	0.5536	-1.0492	0.8952	0.9461
Mean	0.5539	-1.0466	0.8960	0.9466
Std.	0.0003	0.0036	0.0012	0.0006

Table 16: GQA-Judgment behavior and affine fit for LLaVA under the WikiText-after-possibly context. Origin accuracy is evaluated without irrelevant context, while context metrics are evaluated after inserting irrelevant context.
Affine Fit	Behavior
Metric	Value	Metric	Value
Slope a	0.336	Origin Acc.	81.3
Intercept b	-1.377	Context Acc.	61.6
R2	0.580	Context F1	40.4
Pearson r	0.762	Yes Rate	14.5
		Flip Rate	32.1

Table 17: AMBER generative evaluation with LLaVA-1.5-7B. Free and Condition denote context-free and context-conditioned generation, respectively. Hal., Cov., Cog., and Len. denote hallucination rate, object coverage, cognitive error, and average output length.
Setting	CHAIR ↓	Cov. ↑	Hal. ↓	Cog. ↓	Len.
Free	7.9	49.4	33.3	4.0	68.1
Condition	3.8	39.3	9.5	0.7	16.1

Why it matters

In real deployments, chatbots and retrieval-augmented systems often mix in unrelated documents or chat history alongside images, and this work shows for the first time that such noise skews visual judgments in a quantifiable, predictable way rather than randomly. That makes it possible to diagnose and partially correct this bias, which matters for building more trustworthy multimodal AI products.

Terms in this paper

MLLM (multimodal large language model) · an AI model that jointly processes images and text to answer questions or generate descriptions
decision margin · the log-probability difference between two candidate answers (e.g., Yes vs. No), indicating how confidently the model favors one
affine transformation · a straight-line formula of the form y=ax+b that scales and shifts an original value
flip rate · the percentage of samples whose predicted answer changes after irrelevant context is added
post-hoc calibration · a correction applied to model outputs after inference, without retraining the model itself

Figures we cannot republish

Figure 5: Flip rate across context-free margin bins. Samples with smaller |m0| are much more likely to change their predictions after adding irrelevant context. This indicates that context-induced flips are concentrated near the original decision boundary.

See the figures in the original paper →

Original abstract (English)

Multimodal large language models (MLLMs) are frequently exposed to auxiliary textual context, the impact of which on visually grounded tasks remains underexplored. In this paper, we investigate the influence of task-irrelevant context by formulating it as a controlled intervention within a binary visual judgment framework. By maintaining an invariant prompt structure while varying auxiliary inputs, we observe that irrelevant text consistently biases model predictions across diverse benchmarks. To move beyond performance metrics, we characterize this sensitivity through a decision margin defined by the log-probability difference between binary candidates. Our analysis reveals a robust geometric regularity: contextconditioned margins follow a consistent affine transformation of their context-free counterparts. This finding demonstrates that irrelevant context does not manifest as unstructured stochastic noise but as a estimable distortion of model preference. We further interpret the fitted affine parameters as metrics for visual commitment preservation and directional answer bias. These findings provide a margin-level diagnostic view of irrelevant-context effects in MLLMs and offer a basis for future studies on noisy-context robustness

Authors · Yinfeng Wang, Zhiyuan Yao, Zheren Fu, Lei Zhang, Zhendong Mao

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Yinfeng Wang et al., arXiv:2608.19208, CC BY 4.0