NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection

arXiv:2608.192122026-08-21

尼泊尔语假新闻检测:只看文字就能追平图文结合模型

研究团队首次为尼泊尔语构建了一个用于检测断章取义式虚假信息的公开基准数据集,这类虚假信息用真实照片配上误导性文字来编造虚假叙事。团队比较了五种图文结合模型与纯文本、纯图像模型,结果发现只用文字的mBERT模型与表现最好的图文结合模型在统计上完全等效,而只用图像的模型准确率接近随机猜测。这说明在当前数据规模下,文字说明本身已包含足够的判别信息。

他们做了什么

团队从尼泊尔真实的虚假信息事件中收集了545组独特的图片来源,每组都配有一条真实说明和一条虚假说明,共1090条图文对,并按五种类型标注:捏造事实、说明错误、时间错位、地理错位、身份错位
两名标注员独立标注后一致性达到kappa 0.84,说明标注质量可靠
研究者训练了五种不同的图文结合模型(结合ResNet-50、ViT等图像识别模型与mBERT、MuRIL等多语言文本理解模型),并设置了纯文本和纯图像的对照模型进行系统比较
表现最好的图文结合模型(ResNet-50+mBERT)与纯文本的mBERT模型都达到94.65%的Macro-F1分数,McNemar统计检验进一步证实两者几乎犯同样的错误,而非互补
纯图像模型的准确率只有33%到50%,接近随机猜测水平;同时扩大训练数据规模带来的性能提升,比让模型结构变得更复杂更加直接有效

Figure 1: Representative pristine and OOC pairs across five typologies. Red boxes highlight manipulated phrases.

TABLE I: Benchmark comparison. Y=Yes, N=No. Typol.=typology labels; Multi.=multilingual; Low-Res.=low-resource.
Benchmark	OOC	Typol.	Multi.	Low-Res.	Size
NewsCLIPpings [2]	Y	N	N	N	71k
COSMOS [3]	Y	N	N	N	200k
VERITE [4]	Y	N	Partial	N	1k
MuMiN [6]	N	N	Y	Partial	22k
NepOOC (ours)	Y	Y	Y	Y	1,090

Figure 2: Data collection and annotation pipeline: acquisition, verification, annotation, and benchmark output.

TABLE II: Source distribution (n=545 unique images).
Source Type	Count	%
Fact-checking organisations	469	86.1%
Social media archives	49	9.0%
Online news portals	27	5.0%
Total	545	100.0%

Figure 3: Five multimodal architectures organised by visual encoder, text encoder, and fusion mechanism. Double border indicates the best-performing multimodal model (ResNet-50+mBERT; 94.65% Macro-F1).

TABLE III: Typology distribution (n=545 OOC samples).
Typology	Count	% of OOC
Fabricated	299	54.9%
Miscaptioned	136	25.0%
Temporal mismatch	56	10.3%
Geographic mismatch	44	8.1%
Identity mismatch	10	1.8%
Total OOC	545	100.0%

Figure 4: Macro-F1 vs. training fraction with standard deviation bands over 5 seeds.

TABLE IV: Dataset statistics.
Attribute	Category	Count (%)
Label	Pristine	545 (50.0%)
OOC	545 (50.0%)
Language	Nepali	856 (78.5%)
English	158 (14.5%)
Code-switched	76 0(7.0%)
Split	Train	754 (69.2%)
Validation	108 0(9.9%)
Test	228 (20.9%)
Typology IAA (Cohen’s κ)	0.84
Binary IAA, non-FC (Cohen’s κ)	0.81

Figure 5: Precision–Recall and ROC curves for all five models on test split (n=228), best seed per model.

TABLE V: Leakage validation: Δ=Accrandom−Acccluster (mean over 5 seeds). All |Δ|<1% indicates no leakage.
Model	Type	Δ (%)	Status
mBERT	Text-only	+0.66	Clean
CNN+LSTM	Multimodal	−0.39	Clean
ViT+TCN	Multimodal	+0.70	Clean
ResNet-50+mBERT	Multimodal	−0.66	Clean
ViT+MuRIL	Multimodal	+0.48	Clean

TABLE VI: Architecture specifications. VE=visual encoder, TE=text encoder, Params=trainable parameters.
Model	VE	TE	Fusion	TE Vocab	Params
CNN+LSTM	5-layer CNN (512)	LSTM (256/dir)	Late concat	119k	∼16M
ViT+TCN	ViT-B/16 (frozen)	TCN (3 blocks)	Cross-attention	119k	∼90M
ResNet-50+mBERT	ResNet-50 (2048→768)	mBERT (768)	Late concat	119k	134M
CLIP	ViT-B/32 (frozen)	CLIP BPE (frozen)	Similarity head	49k	∼10M (head)
ViT+MuRIL	ViT-B/16 (frozen)	MuRIL+LoRA (r=8)	Cross-attention	197k	∼2.5M (LoRA)

TABLE VII: Training configuration. LR=learning rate, WD=weight decay, BS=batch size, EP=epochs, Pat.=early-stopping patience. †Adaptive LR: 1e−4 at ≤50%, 5e−5 at >50%. ‡StepLR: step 10, γ=0.5.
Model	Opt.	LR	WD	BS	EP	Pat.	Sched.
CNN+LSTM	Adam	1e−4	1e−5	32	80	10	StepLR‡
ViT+TCN	AdamW	5e−5	1e−4	32	100	10	Cos+WU
ResNet +mBERT	AdamW	vis: 1e−4 txt: 2e−5	1e−4	32	50	10	Cos+WU
CLIP	AdamW	†	0.05	8	100	12	—
ViT+MuRIL	AdamW	†	0.05	8	100	12	Cos+WU

TABLE VIII: Main results on test split (n=228, mean±std over 5 seeds).
Model	Type	Acc. (%)	Macro-F1 (%)	AUC
mBERT	Text-only	94.65±0.20	94.65±0.20	0.9697±0.0126
MuRIL	Text-only	94.38±0.35	94.38±0.35	0.9567±0.0158
ResNet-50+mBERT	Multimodal	94.65±0.20	94.65±0.20	0.9662±0.0142
ViT+MuRIL	Multimodal	93.33±0.37	93.33±0.37	0.9505±0.0057
ViT+TCN	Multimodal	92.11±1.35	92.10±1.36	0.9616±0.0064
CNN+LSTM	Multimodal	78.16±10.97	78.15±10.97	0.8548±0.1203
CLIP	Multimodal	69.39±0.72	69.00±0.72	0.7127±0.0028

TABLE IX: McNemar’s test with Yates correction (α=0.05). See text (§5.1) for comparison pairs and discordant counts.
Model A	Model B	Seed	Disc.	b	c	χ2	p	Sig.
text-only mBERT	ResNet-50+mBERT	042	1	0	1	0.000	1.000	×
123	0	0	0	0.000	1.000	×
456	0	0	0	0.000	1.000	×
789	2	2	0	0.500	0.480	×
2024	0	0	0	0.000	1.000	×
Summary	0.6 mean			median p=1.000; 0/5 sig.
ViT+MuRIL	ResNet-50+mBERT	042	1	0	1	0.000	1.000	×
123	0	0	0	0.000	1.000	×
456	3	1	2	0.000	1.000	×
789	1	0	1	0.000	1.000	×
2024	5	0	5	3.200	0.074	×
Summary	2.0 mean			median p=1.000; 0/5 sig.
text-only MuRIL	text-only mBERT	042	2	1	1	0.500	0.480	×
123	3	1	2	0.000	1.000	×
456	0	0	0	0.000	1.000	×
789	5	0	5	3.200	0.074	×
2024	1	0	1	0.000	1.000	×
Summary	2.2 mean			median p=1.000; 0/5 sig.

TABLE X: OOC-class metrics at 100% training data (mean±std over 5 seeds).
Model	Macro-F1 (%)	OOC Prec. (%)	OOC Rec. (%)	OOC F1 (%)
CNN+LSTM	78.15±10.97	79.23±11.27	76.32±11.42	77.72±11.19
ViT+TCN	92.10±1.36	92.20±2.81	92.11±1.96	92.11±1.24
CLIP	69.00±0.72	74.97±1.34	58.25±1.33	65.54±0.87
ResNet-50+mBERT	94.65±0.20	95.37±0.38	93.86±0.00†	94.61±0.19
ViT+MuRIL	93.33±0.37	94.14±1.22	92.46±1.71	93.27±0.43

TABLE XI: Scaling results: Macro-F1 (mean±std over 5 seeds per fraction).
Model	25%	50%	75%	100%
CNN+LSTM	55.22±4.80	61.65±4.99	67.67±9.51	78.15±10.97
ViT+TCN	69.25±11.31	89.12±1.05	90.87±2.57	92.10±1.36
CLIP	62.01±1.25	63.45±2.27	66.14±1.42	69.00±0.72
ResNet-50+mBERT	93.15±0.91	93.77±1.10	94.38±0.79	94.65±0.20
ViT+MuRIL	90.61±0.67	92.72±0.67	93.33±0.72	93.33±0.37

TABLE XII: Modality ablation: Macro-F1 at 100% training data (mean±std over 5 seeds).
Model	Text-only (%)	Image-only (%)	Multimodal (%)
CNN+LSTM	74.91±16.80	49.80±0.27	78.15±10.97
ViT+TCN	90.87±1.80	33.33±0.00	92.10±1.36
CLIP	67.48±1.06	49.20±0.48	69.00±0.72
ResNet+mBERT	94.65±0.20	49.00±0.96	94.65±0.20
ViT+MuRIL	93.95±0.84	49.83±0.18	93.33±0.37

TABLE XIII: Per-seed mean confusion matrices on test split (n=228; 114 Pristine, 114 OOC).
Model	Actual	Pred: Pristine	Pred: OOC
CNN+LSTM	Pristine	91.2	22.8
OOC	27.0	87.0
ViT+TCN	Pristine	105.0	9.0
OOC	9.0	105.0
CLIP	Pristine	91.8	22.2
OOC	47.6	66.4
ResNet-50+mBERT	Pristine	108.8	5.2
OOC	7.0	107.0
ViT+MuRIL	Pristine	107.4	6.6
OOC	8.6	105.4

TABLE XIV: Per-typology OOC Macro-F1 at 100% training data (mean±std over 5 seeds). n=OOC test instances per typology.
Typology	n	CNN+LSTM	ViT+TCN	CLIP	ResNet-50+mBERT	ViT+MuRIL
Fabricated	62	83.93±7.57	96.14±0.99	74.49±1.06	95.80±1.00	95.97±0.39
Miscaptioned	28	87.92±8.92	95.09±2.23	68.53±1.70	98.18±0.00†	95.89±1.37
Other Mismatches‡	24	89.01±9.50	91.43±4.21	74.81±4.42	97.79±0.00†	96.38±3.16

为什么重要

此前的虚假信息检测研究几乎都集中在英语场景,尼泊尔语这样的资源匮乏语言此前没有公开的评测基准,这项工作填补了这一空白并开源了数据和代码。它还挑战了图文结合模型一定优于纯文本模型的普遍假设,提醒研究者在资源有限的语言场景中评测多模态模型时,必须把纯文本基线作为重要对照。

本文术语

断章取义式虚假信息(OOC) · 不篡改图片本身,而是给真实图片配上误导性文字说明来制造虚假叙事的一种造假方式
mBERT · 在104种语言的文本上预训练的多语言语言模型,这里用来理解图片说明文字
ResNet-50 · 一种广泛用于提取图像特征的卷积神经网络结构
Macro-F1 · 对真实和虚假两个类别的精确率与召回率进行平均计算得到的综合性能指标
McNemar检验 · 一种统计方法,用来判断两个分类模型的正确与错误预测模式是否存在显著差异
科恩卡帕系数(Cohen's kappa) · 衡量多名标注者标注结果一致程度、并排除偶然因素影响的统计指标

论文原文摘要（英文）

Out-of-context (OOC) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making detection a problem of multimodal alignment rather than image forensics. Despite the prevalence and consequences of OOC misinformation in Nepal, no public benchmark exists for Nepali. We introduce NepOOC, the first publicly available Nepali-dominant multilingual OOC benchmark, comprising 1,090 image-caption pairs (545 pristine, 545 OOC) annotated across five typologies (fabricated, miscaptioned, temporal mismatch, geographic mismatch, identity mismatch) with inter-annotator agreement kappa = 0.84. Systematic evaluation of five multimodal architectures alongside text-only and image-only baselines reveals that caption semantics appear sufficient for strong performance at the current dataset scale. A text-only mBERT model achieves 94.65+/-0.20% Macro-F1, statistically equivalent to the best multimodal system (ResNet-50+mBERT, 94.65+/-0.20%; McNemar median p = 1.000, 0/5 seeds significant at alpha = 0.05). Image-only models perform near chance (33-50%), while training-size scaling suggests that dataset expansion is a more direct path to progress than architectural sophistication or regional specialisation.

作者 · Sanjeev Khatiwada

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Sanjeev Khatiwada et al., arXiv:2608.19212, CC BY 4.0

NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道