UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
arXiv:2608.185042026-08-20
跨图像视频文档的检索AI学会把候选项两两比对,说清楚为什么是这个答案
多模态检索系统通常把问题和每个候选项分开处理,只计算相似度打分,这样很难区分正确答案和长得很像的错误答案。UMER让模型把问题和候选项配对放在一起进行比较推理,明确指出哪里匹配哪里不匹配,同时在同一个模型里联合训练快速的向量嵌入检索和精细的成对排序打分,让两者互相取长补短。在包含78个任务的MMEB-V2基准测试中,该方法取得了优于此前方法的成绩,并且可以根据需要在速度和精度之间灵活权衡。
他们做了什么
- 问题所在:现有的思维链(CoT)推理方法只分别描述问题和候选项各自的内容,无法生成用来区分正确答案与相似错误答案(难负样本)的证据。
- 解决方案:UMER训练模型进行'成对判别式推理',把正样本候选项和容易混淆的难负样本候选项与问题一起呈现,让模型明确比较两者的匹配证据和差异证据。
- 架构设计:在单个多模态大语言模型(MLLM)内部同时训练用于大规模向量检索的嵌入表示和直接对问题候选对打分的判别式排序功能,并通过'互补式相互蒸馏(CMD)'机制,只在结果可信时让两个功能互相传递知识来共同优化。
- 数据构建:使用Qwen3.5-9B为每个正负样本对生成包含查询意图和目标观察的推理内容,再用一个小型验证模型(Qwen3.5-0.8B)检查仅凭这段推理能否正确判断出真实标签,以此过滤低质量数据。
- 结果表现:在包含78个任务的MMEB-V2基准上,UMER的混合模式总体得分达到65.5,优于对比方法;其纯嵌入检索模式在查询速度上比此前的推理式方法UME-R1快达118.6倍。
Table 1: Main results on MMEB-V2. The best and second-best scores in each column are in bold and underlined, respectively.| Model | Image | Video | VisDoc | All |
|---|
| CLS | QA | RET | GD | Overall | CLS | QA | RET | MRET | Overall | VDRv1 | VDRv2 | VR | OOD | Overall | |
| # of Datasets | 10 | 10 | 12 | 4 | 36 | 5 | 5 | 5 | 3 | 18 | 10 | 4 | 6 | 4 | 24 | 78 |
| Baseline Models |
| GME | 54.4 | 29.9 | 66.9 | 55.5 | 51.9 | 34.9 | 42.0 | 25.6 | 32.4 | 33.9 | 86.1 | 54.0 | 82.5 | 43.1 | 72.7 | 54.1 |
| VLM2Vec | 58.7 | 49.3 | 65.0 | 72.9 | 59.7 | 33.4 | 30.5 | 20.6 | 33.0 | 29.0 | 49.8 | 13.5 | 51.8 | 33.5 | 41.6 | 47.0 |
| VLM2Vec-V2 | 62.9 | 56.3 | 69.5 | 77.3 | 64.9 | 39.3 | 34.3 | 28.8 | 38.5 | 34.9 | 75.5 | 44.9 | 79.4 | 39.4 | 65.4 | 58.0 |
| DUME | 59.3 | 55.0 | 66.3 | 78.0 | 62.5 | 37.7 | 46.6 | 17.1 | 30.0 | 33.2 | 67.6 | 43.3 | 47.1 | 33.8 | 52.8 | 52.7 |
| BToks | 64.3 | 59.8 | 68.8 | 77.4 | 66.0 | 43.7 | 47.0 | 33.0 | 33.6 | 39.9 | 71.1 | 38.6 | 81.3 | 38.1 | 62.7 | 59.0 |
| UME-R1 | 64.8 | 62.8 | 67.6 | 77.2 | 66.6 | 44.3 | 51.2 | 32.9 | 39.7 | 42.2 | 72.4 | 46.2 | 79.2 | 37.2 | 63.9 | 60.1 |
| PLUME | 66.5 | 59.2 | 67.6 | 79.7 | 66.3 | 45.0 | 52.3 | 33.5 | 46.7 | 44.1 | 72.1 | 49.8 | 78.1 | 57.4 | 67.5 | 61.6 |
| RIME | 67.9 | 64.4 | 69.8 | 82.1 | 69.1 | 48.0 | 52.1 | 33.6 | 39.2 | 43.7 | 76.4 | 51.4 | 81.7 | 63.9 | 71.4 | 64.1 |
| Ours |
| UMER-E | 64.4 | 64.9 | 70.2 | 78.4 | 68.0 | 43.0 | 50.6 | 34.7 | 32.8 | 41.1 | 76.1 | 50.1 | 82.8 | 68.3 | 72.2 | 63.1 |
| UMER-R | 64.9 | 68.4 | 69.7 | 74.1 | 68.5 | 51.4 | 51.6 | 35.3 | 33.3 | 44.0 | 75.1 | 48.3 | 84.6 | 67.9 | 71.8 | 63.9 |
| UMER-H | 66.1 | 68.9 | 71.7 | 80.8 | 70.4 | 50.6 | 52.8 | 37.7 | 35.0 | 45.0 | 78.0 | 50.5 | 85.0 | 68.8 | 73.6 | 65.5 |
Table 2: Controlled ablations of pair-aware CoT, ranking supervision and complementary mutual distillation (CMD). The best and second-best scores in each column are in bold and underlined, respectively.| Configuration | Embedding mode (UMER-E) | Ranking mode (UMER-R) | Hybrid mode (UMER-H) |
|---|
| Image | Video | VisDoc | All | Image | Video | VisDoc | All | Image | Video | VisDoc | All |
| Embedding only | 66.5 | 37.8 | 69.2 | 60.7 | – | – | – | – | – | – | – | – |
| + pair-aware CoT, w/o ranking losses | 68.2 | 41.1 | 71.2 | 62.9 | 44.4 | 31.8 | 57.1 | 45.4 | 68.6 | 42.2 | 71.3 | 63.3 |
| + ranking losses, w/o CoT | 68.0 | 40.2 | 69.3 | 62.0 | 68.6 | 43.1 | 69.4 | 63.0 | 70.8 | 45.1 | 71.2 | 65.0 |
| + pair-aware CoT and ranking losses | 67.9 | 39.6 | 71.1 | 62.4 | 67.2 | 44.2 | 72.1 | 63.4 | 70.3 | 44.9 | 73.4 | 65.4 |
| Pair-aware model, w/o CMD | 67.9 | 39.6 | 71.1 | 62.4 | 67.2 | 44.2 | 72.1 | 63.4 | 70.3 | 44.9 | 73.4 | 65.4 |
| + ranking → embedding only | 67.9 | 40.4 | 71.5 | 62.7 | 68.5 | 43.3 | 70.9 | 63.4 | 70.6 | 44.7 | 72.7 | 65.3 |
| + embedding → ranking only | 67.8 | 40.4 | 70.1 | 62.2 | 69.2 | 43.9 | 71.1 | 63.9 | 70.8 | 45.1 | 72.6 | 65.5 |
| + always-on bidirectional CMD | 68.0 | 40.2 | 71.1 | 62.6 | 68.5 | 43.6 | 71.0 | 63.6 | 70.7 | 45.3 | 73.0 | 65.5 |
| + selective bidirectional CMD (full) | 68.0 | 41.1 | 72.2 | 63.1 | 68.5 | 44.0 | 71.8 | 63.9 | 70.4 | 45.0 | 73.6 | 65.5 |
Table 3: Accuracy–efficiency on MMEB-V2 using one NVIDIA A100 GPU (batch size=1). UMER-H reuses the UMER-E index; 0. denotes no extra indexing cost.| Model | K | Score ↑ | Reasoning Tokens | Query Latency | Indexing Time |
|---|
| (/query) | (s/query) ↓ | (s/candidate) ↓ |
| UME-R1 | – | 60.1 | 352 | 9.963 | 11.755 |
| PLUME | – | 61.6 | 8 | 0.329 | 0.366 |
| UMER-E | – | 63.1 | 0 | 0.084 | 0.118 |
| UMER-H | 3 | 64.8 | 410 | 13.566 | 0. |
| 5 | 65.5 | 665 | 19.880 |
| 10 | 66.0 | 1305 | 42.764 |
| 20 | 66.1 | 2508 | 81.726 |
Table 1: Implementation and evaluation settings for UMER.| Setting | Value |
|---|
| Model and optimization |
| Initialization | Qwen2-VL-2B-Instruct. |
| Embedding tokens | Four learnable tokens per input (M=4). |
| Embedding temperature | 0.02. |
| Objective weights | λcot=0.2, λbce=0.1, λmargin=0.2, and λcmd=0.2. |
| Optimizer and schedule | AdamW; per-device batch size 64; one accumulation step; linear schedule with peak rate 5×10−5, 100 warmup steps and 10,000 maximum steps. |
| Adaptation | LoRA rank 16, scaling 64 and dropout 0.1 on attention/MLP projections and the language-model head; visual encoder frozen. |
| Precision and preprocessing | BF16 with FlashAttention-2; maximum image pixels 2,359,296. |
| Inference |
| Candidate selection | Retrieve with the embedding branch and rerank the top K=5 candidates. |
| Ranking generation | At most 256 newly generated tokens per candidate. |
| Hybrid fusion | Separately z-score normalize embedding and ranking scores over the five candidates. |
| Ranking-score weight | α=2.0. |
| Evaluation and reporting |
| Benchmark | MMEB-V2: 78 datasets across image, video and visual-document modalities. |
| Benchmark versions | Corrected ViDoSeek-page and MMLongBench-page versions. |
| Metrics | Hit@1 for image and video tasks; NDCG@5 for visual-document tasks. |
| Aggregation | Unweighted macro averages over datasets, including all 78 tasks for All. |
Table 4: Ranking-score-weight sensitivity of UMER-H at K=5, with candidates, decoding configuration, and normalization fixed.| Ranking-score weight | 0.5 | 0.75 | 1 | 1.5 | 2 | 3 | 5 |
|---|
| UMER-H | 64.7 | 65.0 | 65.2 | 65.5 | 65.5 | 65.4 | 65.3 |
Table 5: Decision overlap of embedding retrieval (E) and pair-aware ranking (R) over 83,530 MMEB-V2 queries. Both, E only, R only, and Neither indicate which branch produces the correct top decision. Each cell is a within-family percentage.| Family | Both | E only | R only | Neither |
|---|
| Reasoning / semantic | 48.8 | 9.0 | 12.1 | 30.1 |
| Content matching | 45.4 | 8.9 | 8.3 | 37.4 |
| All | 47.2 | 9.0 | 10.3 | 33.5 |
为什么重要
这为需要在海量图像、视频、文档中快速筛选候选项、同时又要准确辨别相似干扰项的实际搜索或推荐系统提供了可落地的方案。该方法还支持根据场景需求在速度和精度之间调节,对实际系统设计具有直接参考价值。
本文术语
- 嵌入(Embedding) · 把文本或图像转换成数字向量,使相似内容在向量空间中彼此靠近的表示方法
- 难负样本(Hard Negative) · 与正确答案在外观或语义上十分相似、容易被误判的错误候选项
- 思维链(Chain-of-Thought, CoT) · 让模型在给出最终答案前先写出中间推理步骤的技术
- 重排序(Reranking) · 对第一轮快速检索出的少量候选项,用更精细的模型再次排序的二阶段检索方式
- 知识蒸馏(Distillation) · 让一个模型或功能的判断结果去训练另一个模型或功能,使其学到相似行为的方法
无法转载的图表
- Figure 1: Two overlooked issues in universal multimodal retrieval. (a) Item-wise self-reflective CoT lacks pair-aware evidence for hard-negative discrimination. (b) Different meta- tasks require different capabilities, with embedding and ranking offering complementary strengths.
- Figure 2: Overview of UMER, a unified multimodal embedding and ranking framework. (a) Query and candidate inputs are first encoded independently through masked branch attention to extract their embeddings, and are then jointly used for pair-aware autoregressive reasoning followed by a ranking token. (b) Multi-task heads support metric embedding learning, generative CoT learning and discriminative ranking learning. (c) Unified co-optimization combines multi-task supervision and complementary mutual distillation to jointly improve embedding and ranking.
- Figure 3: Embedding separation under item-wise and pair-aware supervision on 3,600 queries from the 36 image tasks of MMEB-V2. (a) Bootstrap distribution of the macro-averaged positive–hard-negative margin. (b) Fraction of queries exceeding margin thresholds.
- Figure 4: Capability specialization and transfer via CMD. (a) Embedding is stronger for content matching, while ranking is stronger for reasoning-intensive relevance judgment. (b) CMD transfers these complementary strengths between the two functions.
在原文中查看图表 →论文原文摘要（英文）
Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-grained semantic reasoning. Recent MLLM-based embedding methods typically derive representations from hidden states, while Chain-of-Thought (CoT) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space. However, exist
作者 · Libiao Chen, Xiyang Liu, Yanheng Wei, Tao Wang, Zhenyu Tang
在 arXiv 阅读
