NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection
arXiv:2608.192122026-08-21
Nepali fake-news detector matches image+text model using text alone
Researchers built the first Nepali-focused benchmark for out-of-context (OOC) misinformation, where real photos are paired with misleading captions. Comparing five multimodal models against text-only and image-only baselines, a text-only mBERT model performed statistically identically to the best image-plus-text model, while image-only models barely beat random guessing. The findings suggest that at this dataset's current size, caption text alone carries almost all the useful signal.
What they did
The team gathered 545 real image-caption pairs from actual Nepali misinformation incidents, each paired with both a genuine (pristine) and a falsified (OOC) caption, for 1,090 total pairs labeled into five manipulation types: fabricated, miscaptioned, temporal mismatch, geographic mismatch, and identity mismatch
Two independent annotators labeled the data with an inter-annotator agreement of kappa 0.84, indicating strong labeling reliability
Five different image-plus-text model architectures (combining visual encoders like ResNet-50 or ViT with text encoders like mBERT or MuRIL) were trained and compared against dedicated text-only and image-only baselines under identical conditions
The best multimodal model (ResNet-50+mBERT) and a text-only mBERT model both scored 94.65% Macro-F1, and a statistical test (McNemar's test) confirmed the two models made nearly identical predictions rather than complementary ones
Image-only models scored only 33-50% (near chance level), and experiments scaling up the amount of training data showed that adding more data improves performance more reliably than making the model architecture more complex
Figure 1: Representative pristine and OOC pairs across five typologies. Red boxes highlight manipulated phrases.
Figure 2: Data collection and annotation pipeline: acquisition, verification, annotation, and benchmark output.
TABLE II: Source distribution (n=545 unique images).
Source Type
Count
%
Fact-checking organisations
469
86.1%
Social media archives
49
9.0%
Online news portals
27
5.0%
Total
545
100.0%
Figure 3: Five multimodal architectures organised by visual encoder, text encoder, and fusion mechanism. Double border indicates the best-performing multimodal model (ResNet-50+mBERT; 94.65% Macro-F1).
TABLE III: Typology distribution (n=545 OOC samples).
Typology
Count
% of OOC
Fabricated
299
54.9%
Miscaptioned
136
25.0%
Temporal mismatch
56
10.3%
Geographic mismatch
44
8.1%
Identity mismatch
10
1.8%
Total OOC
545
100.0%
Figure 4: Macro-F1 vs. training fraction with standard deviation bands over 5 seeds.
TABLE IV: Dataset statistics.
Attribute
Category
Count (%)
Label
Pristine
545 (50.0%)
OOC
545 (50.0%)
Language
Nepali
856 (78.5%)
English
158 (14.5%)
Code-switched
76 0(7.0%)
Split
Train
754 (69.2%)
Validation
108 0(9.9%)
Test
228 (20.9%)
Typology IAA (Cohen’s κ)
0.84
Binary IAA, non-FC (Cohen’s κ)
0.81
Figure 5: Precision–Recall and ROC curves for all five models on test split (n=228), best seed per model.
TABLE V: Leakage validation: Δ=Accrandom−Acccluster (mean over 5 seeds). All |Δ|<1% indicates no leakage.
TABLE VII: Training configuration. LR=learning rate, WD=weight decay, BS=batch size, EP=epochs, Pat.=early-stopping patience. †Adaptive LR: 1e−4 at ≤50%, 5e−5 at >50%. ‡StepLR: step 10, γ=0.5.
Model
Opt.
LR
WD
BS
EP
Pat.
Sched.
CNN+LSTM
Adam
1e−4
1e−5
32
80
10
StepLR‡
ViT+TCN
AdamW
5e−5
1e−4
32
100
10
Cos+WU
ResNet +mBERT
AdamW
vis: 1e−4 txt: 2e−5
1e−4
32
50
10
Cos+WU
CLIP
AdamW
†
0.05
8
100
12
—
ViT+MuRIL
AdamW
†
0.05
8
100
12
Cos+WU
TABLE VIII: Main results on test split (n=228, mean±std over 5 seeds).
Model
Type
Acc. (%)
Macro-F1 (%)
AUC
mBERT
Text-only
94.65±0.20
94.65±0.20
0.9697±0.0126
MuRIL
Text-only
94.38±0.35
94.38±0.35
0.9567±0.0158
ResNet-50+mBERT
Multimodal
94.65±0.20
94.65±0.20
0.9662±0.0142
ViT+MuRIL
Multimodal
93.33±0.37
93.33±0.37
0.9505±0.0057
ViT+TCN
Multimodal
92.11±1.35
92.10±1.36
0.9616±0.0064
CNN+LSTM
Multimodal
78.16±10.97
78.15±10.97
0.8548±0.1203
CLIP
Multimodal
69.39±0.72
69.00±0.72
0.7127±0.0028
TABLE IX: McNemar’s test with Yates correction (α=0.05). See text (§5.1) for comparison pairs and discordant counts.
Model A
Model B
Seed
Disc.
b
c
χ2
p
Sig.
text-only mBERT
ResNet-50+mBERT
042
1
0
1
0.000
1.000
×
123
0
0
0
0.000
1.000
×
456
0
0
0
0.000
1.000
×
789
2
2
0
0.500
0.480
×
2024
0
0
0
0.000
1.000
×
Summary
0.6 mean
median p=1.000; 0/5 sig.
ViT+MuRIL
ResNet-50+mBERT
042
1
0
1
0.000
1.000
×
123
0
0
0
0.000
1.000
×
456
3
1
2
0.000
1.000
×
789
1
0
1
0.000
1.000
×
2024
5
0
5
3.200
0.074
×
Summary
2.0 mean
median p=1.000; 0/5 sig.
text-only MuRIL
text-only mBERT
042
2
1
1
0.500
0.480
×
123
3
1
2
0.000
1.000
×
456
0
0
0
0.000
1.000
×
789
5
0
5
3.200
0.074
×
2024
1
0
1
0.000
1.000
×
Summary
2.2 mean
median p=1.000; 0/5 sig.
TABLE X: OOC-class metrics at 100% training data (mean±std over 5 seeds).
Model
Macro-F1 (%)
OOC Prec. (%)
OOC Rec. (%)
OOC F1 (%)
CNN+LSTM
78.15±10.97
79.23±11.27
76.32±11.42
77.72±11.19
ViT+TCN
92.10±1.36
92.20±2.81
92.11±1.96
92.11±1.24
CLIP
69.00±0.72
74.97±1.34
58.25±1.33
65.54±0.87
ResNet-50+mBERT
94.65±0.20
95.37±0.38
93.86±0.00†
94.61±0.19
ViT+MuRIL
93.33±0.37
94.14±1.22
92.46±1.71
93.27±0.43
TABLE XI: Scaling results: Macro-F1 (mean±std over 5 seeds per fraction).
Model
25%
50%
75%
100%
CNN+LSTM
55.22±4.80
61.65±4.99
67.67±9.51
78.15±10.97
ViT+TCN
69.25±11.31
89.12±1.05
90.87±2.57
92.10±1.36
CLIP
62.01±1.25
63.45±2.27
66.14±1.42
69.00±0.72
ResNet-50+mBERT
93.15±0.91
93.77±1.10
94.38±0.79
94.65±0.20
ViT+MuRIL
90.61±0.67
92.72±0.67
93.33±0.72
93.33±0.37
TABLE XII: Modality ablation: Macro-F1 at 100% training data (mean±std over 5 seeds).
Model
Text-only (%)
Image-only (%)
Multimodal (%)
CNN+LSTM
74.91±16.80
49.80±0.27
78.15±10.97
ViT+TCN
90.87±1.80
33.33±0.00
92.10±1.36
CLIP
67.48±1.06
49.20±0.48
69.00±0.72
ResNet+mBERT
94.65±0.20
49.00±0.96
94.65±0.20
ViT+MuRIL
93.95±0.84
49.83±0.18
93.33±0.37
TABLE XIII: Per-seed mean confusion matrices on test split (n=228; 114 Pristine, 114 OOC).
Model
Actual
Pred: Pristine
Pred: OOC
CNN+LSTM
Pristine
91.2
22.8
OOC
27.0
87.0
ViT+TCN
Pristine
105.0
9.0
OOC
9.0
105.0
CLIP
Pristine
91.8
22.2
OOC
47.6
66.4
ResNet-50+mBERT
Pristine
108.8
5.2
OOC
7.0
107.0
ViT+MuRIL
Pristine
107.4
6.6
OOC
8.6
105.4
TABLE XIV: Per-typology OOC Macro-F1 at 100% training data (mean±std over 5 seeds). n=OOC test instances per typology.
Typology
n
CNN+LSTM
ViT+TCN
CLIP
ResNet-50+mBERT
ViT+MuRIL
Fabricated
62
83.93±7.57
96.14±0.99
74.49±1.06
95.80±1.00
95.97±0.39
Miscaptioned
28
87.92±8.92
95.09±2.23
68.53±1.70
98.18±0.00†
95.89±1.37
Other Mismatches‡
24
89.01±9.50
91.43±4.21
74.81±4.42
97.79±0.00†
96.38±3.16
Why it matters
OOC misinformation research has largely focused on English, leaving no public benchmark for Nepali despite documented real-world harm from such content; this work fills that gap with a dataset and open code release. It also challenges the common assumption that combining images and text always beats text alone, which has practical implications for how researchers should evaluate multimodal misinformation detectors in low-resource language settings.
Terms in this paper
Out-of-Context (OOC) misinformation · False narratives built by pairing an authentic, unaltered image with a misleading caption rather than by editing the image itself
mBERT · A multilingual language model pretrained on text from 104 languages, used here to read captions
ResNet-50 · A widely used convolutional neural network for extracting visual features from images
Macro-F1 · A performance score that averages precision and recall equally across both the real and fake classes
McNemar's test · A statistical test that checks whether two classifiers' correct/incorrect predictions differ significantly
Cohen's kappa · A statistic measuring how much human annotators agree beyond what would happen by chance
Original abstract (English)
Out-of-context (OOC) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making detection a problem of multimodal alignment rather than image forensics. Despite the prevalence and consequences of OOC misinformation in Nepal, no public benchmark exists for Nepali. We introduce NepOOC, the first publicly available Nepali-dominant multilingual OOC benchmark, comprising 1,090 image-caption pairs (545 pristine, 545 OOC) annotated across five typologies (fabricated, miscaptioned, temporal mismatch, geographic mismatch, identity mismatch) with inter-annotator agreement kappa = 0.84. Systematic evaluation of five multimodal architectures alongside text-only and image-only baselines reveals that caption semantics appear sufficient for strong performance at the current dataset scale. A text-only mBERT model achieves 94.65+/-0.20% Macro-F1, statistically equivalent to the best multimodal system (ResNet-50+mBERT, 94.65+/-0.20%; McNemar median p = 1.000, 0/5 seeds significant at alpha = 0.05). Image-only models perform near chance (33-50%), while training-size scaling suggests that dataset expansion is a more direct path to progress than architectural sophistication or regional specialisation.
