Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

arXiv:2608.199422026-08-21

When text and image seem to agree but secretly clash, that mismatch is sarcasm — and this AI learns to spot it

AI systems that detect sarcasm in posts combining text and images often struggle when the words and picture look superficially compatible but actually contradict each other in meaning. This paper builds a framework that dynamically decides, case by case, how much to trust text versus image, and adds a contrastive learning trick that treats surface-level agreement in sarcastic posts as a trap rather than evidence of sincerity. Tested on the MMSD and MMSD2.0 benchmarks, the method consistently beat strong existing baselines.

Original abstract (English)

Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.

Authors · Hao Guo, Subin Huang, Junjie Chen, Zhifa Geng, Sanmin Liu, Chao Kong

Figures: Hao Guo et al., arXiv:2608.19942, arxiv-nonexclusive