Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correction

arXiv:2608.199712026-08-21

文本信息缺失或损坏时,这个AI不靠一次性猜测,而是反复修正猜测结果,从而更准确地判断情绪

现实场景中的多模态情绪分析经常要处理缺失或受损的语音、视频、文本信息,这会拖累识别准确率。以往方法在文本信息不足时,会用音频视频信息生成一个替代表示(代理),但只生成一次就直接投入使用,初期误差会一路传导下去。这篇论文提出让这个代理表示分多个阶段逐步修正,在MOSI、MOSEI、SIMS三个数据集上都比此前最强基线LNLN表现更稳定、更好。

论文原文摘要（英文）

Multimodal sentiment analysis aims to infer affective states by integrating language, visual, and acoustic cues. However, real-world multimodal inputs are often incomplete or corrupted, which can weaken cross-modal complementarity and introduce misleading information into downstream fusion. Existing proxy-based methods for incomplete MSA commonly rely on one-shot proxy construction to compensate for degraded language information, but the generated proxy may be coarse or unreliable at initialization. Prematurely injecting such a proxy into multimodal reasoning can propagate initial errors and compromise sentiment prediction. To address this limitation, we propose an iterative proxy correction framework for robust incomplete MSA. Our method constructs a language-oriented proxy from non-language modalities and progressively refines it under multimodal context through gated residual correction. The corrected proxy is then adaptively fused with the observed language representation according to an estimated language reliability score, allowing the model to balance proxy-based compensation and trustworthy linguistic evidence. In addition, we introduce a stage-wise latent correction objective that uses the complete language representation as a training-time semantic anchor to stabilize the proxy refinement trajectory. Extensive experiments on MOSI, MOSEI, and SIMS under diverse missing-modality settings demonstrate that the proposed framework consistently outperforms competitive baselines and achieves robust sentiment prediction under incomplete inputs.

作者 · Zhifa Geng, Subin Huang, Hao Guo, Junjie Chen, Sanmin Liu, Chao Kong

图片来源： Zhifa Geng et al., arXiv:2608.19971, arxiv-nonexclusive