每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

arXiv:2608.194082026-08-21

AI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤

在线策略蒸馏(On-Policy Distillation, OPD)让学生模型根据自己生成的答案对照老师模型打分学习,但这种打分方式有个盲点:即使学生的推理步骤正在真正接近正确答案,只要写法和老师不一样就会被扣分。研究团队提出R2-OPD,专门独立衡量每一步推理是否真的在接近答案,当老师打分和真实进展方向相矛盾时,就把这一段的扣分信号过滤掉。在DeepSeek-R1-Distill-Qwen-1.5B上的实验中,相比标准OPD,平均准确率(avg@4)提升2.51分,四次尝试中至少答对一次的比例(pass@4)提升4.46分。

他们做了什么

  1. 研究发现OPD按token逐一对照老师模型分布打分的方式,并不总能准确反映学生是否真的在朝正确答案推进
  2. 为独立衡量真实进展,团队从每个中间推理节点出发做Neval=8次模拟续写,估算学生从这一步能答对的概率,由此计算出真正的进展分数(process reward)
  3. 将进展方向一致的相邻推理片段合并(sign-consistent merging)以降低噪声,再比较进展分数排序与老师相似度分数排序是否矛盾,对矛盾片段的老师监督信号进行屏蔽(masking)
  4. 在以JustRL为老师模型的DeepSeek-R1-Distill-Qwen-1.5B实验中,R2-OPD相比标准OPD平均准确率(avg@4)提升2.51分,pass@4提升4.46分,在AIME数学竞赛题上提升尤为明显
  5. 该方法在Qwen3-1.7B学生模型搭配e3-1.7B老师模型的组合上同样有效,说明方法能跨模型系列迁移
Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).
Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).
Table 1: OPD performance on DeepSeek-R1-Distill-Qwen-1.5B with JustRL as the teacher model.
MethodAIME 24AIME 25OlympiadAvg.
avg@4pass@4avg@4pass@4avg@4pass@4avg@4pass@4
Student22.5043.3323.3336.6743.1958.3129.6746.10
Teacher41.6756.6730.8343.4453.2868.4341.9256.18
OPD (1)28.3350.0022.5030.0046.8662.1032.5547.37
E-OPD (17)18.3336.6712.5023.3349.9164.5626.9141.52
TIP-OPD (36)17.5033.3311.6720.0048.2462.8325.8038.72
IW-OPD (34)20.8336.6717.5026.6743.5959.0927.3140.81
Uni-OPD (13)20.0043.3319.1726.6753.1669.9730.7846.66
R2-OPD (Ours)32.5056.6725.8336.6746.8662.1935.0651.83
Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment (J9) receives a smaller KL divergence under Equation (9) than segments that derive or recover the correct solution, respectively(J2 and J10).
Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment (J9) receives a smaller KL divergence under Equation (9) than segments that derive or recover the correct solution, respectively(J2 and J10).
Table 2: Transfer performance of R2-OPD on Qwen3-1.7B with e3-1.7B as the teacher model.
DatasetBaseOPDR2-OPD
avg@4pass@4avg@4pass@4avg@4pass@4
AIME 2424.1730.0022.5036.6725.0040.00
AIME 2518.7520.0027.5033.3325.8336.67
Olympiad50.1063.8753.8267.0954.3167.91
Avg.31.0137.9634.6145.7035.0448.19
Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.
Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.
Table 3: Effect of sign-consistent segment merging on avg@4 accuracy. Δ denotes the improvement of the full R2-OPD over its no-merging variant.
DatasetNo MergeR2-OPDΔ
AIME 2417.532.5015.0
AIME 2511.6725.8314.16
Olympiad47.846.86-0.96
Figure 4: Sensitivity of R2-OPD to the masking ratio q. A moderate ratio of q=30 provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.
Figure 4: Sensitivity of R2-OPD to the masking ratio q. A moderate ratio of q=30 provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.
Table 4: Optimization, process-reward estimation, and filtering settings used in all R2-OPD experiments.
SettingValue
Optimization and sequence settings
Training epochs1
OptimizerAdamW
Learning rate5×10−6
Global batch size64
Maximum prompt length1,024 tokens
Maximum response length7,168 tokens
KL support sizeStudent top-16 (H=16)
Process-reward estimation
Rollouts per evaluated boundaryNeval=8
Sampling temperature0.7
Top-k50
Top-p1.0
Maximum rollout length300 tokens
Segment filtering
Minimum sentences between boundariesSmin=3
Minimum merged segmentsnmin=3
Segment masking ratioq=30%
Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.
Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.
Table 5: Eligibility checks and conservative fallback actions used by R2-OPD.
ConditionAction
The string-level pre-check does not find gi in yi, or segmentation yields fewer than two segments.Skip process-reward rollouts and retain all token-level OPD supervision.
The response has fewer than max⁡(3,nmin) merged segments.Do not apply segment masking.
No strict adjacent pair is available after process-reward ranking.Do not apply segment masking.
All candidate segments have zero inconsistency score.Do not apply segment masking.
At least one candidate has a positive inconsistency score.Mask up to the response-specific budget; retain every other token.
Table 6: Complete lexicon of discourse markers used to propose reasoning-segment boundaries.
CategoryMatched markers
Reconsiderationwait, hold on, let me reconsider, hmm
Correctionactually
Verificationlet me check
Alternative reasoningalternatively

为什么重要

这项研究揭示了传统师生蒸馏训练的一个结构性缺陷:只因写法和老师不同就压制学生本可能正确、有创意的推理路径。对于希望更便宜、更有效地训练数学或编程等推理型AI模型的团队,这提供了可直接借鉴的训练改进思路。

本文术语

  • 在线策略蒸馏(On-Policy Distillation, OPD) · 让学生模型用自己生成的答案接受老师模型打分来学习的一种知识蒸馏方法
  • 反向KL散度(reverse KL divergence) · 衡量学生模型概率分布与老师模型差异程度的指标,常被用作训练信号
  • 进展分数(process reward) · 衡量某一步推理相比前一步,答对最终问题概率提升了多少的分数
  • 方向一致合并(sign-consistent merging) · 把进展方向相同的相邻推理片段合并起来,以降低测量噪声
  • avg@4 / pass@4 · avg@4是四次尝试的平均准确率,pass@4是四次尝试中至少答对一次的比例

论文原文摘要（英文）

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

作者 · Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Chen Yang et al., arXiv:2608.19408, arxiv-nonexclusive