Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

arXiv:2608.199422026-08-21

텍스트와 이미지가 겉으로는 잘 맞아도 속뜻이 어긋나면 비꼬는 말이다, AI가 그 함정을 잡아낸다

글과 사진을 함께 올리는 SNS 게시물에서 비꼼(풍자)을 감지하는 AI는 표면적으로 말과 그림이 잘 어울려 보여도 실제로는 반대 의미를 담고 있는 경우를 놓치기 쉬웠다. 이 연구는 텍스트와 이미지 중 어느 쪽이 더 결정적인 단서인지 게시물마다 다르게 판단하는 장치와, 겉보기 일치가 오히려 함정임을 학습시키는 대조 학습 기법을 결합한 프레임워크를 제안한다. MMSD와 MMSD2.0이라는 두 벤치마크 실험에서 기존 강력한 비교 모델들보다 꾸준히 높은 성능을 보였다.

논문 원문 초록 (영문)

Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.

저자 · Hao Guo, Subin Huang, Junjie Chen, Zhifa Geng, Sanmin Liu, Chao Kong

그림 출처: Hao Guo et al., arXiv:2608.19942, arxiv-nonexclusive