매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

arXiv:2608.194082026-08-21

AI가 선생님 모델을 따라 배우다가, 정답에 다가가는 '좋은 생각'까지 억누르는 문제를 잡아낸다

학생 모델이 스스로 만든 답변을 선생님 모델과 비교해 학습시키는 On-Policy Distillation(OPD) 방식은, 학생이 선생님과 다른 방식으로 생각해도 무조건 감점을 주는 맹점이 있다. 연구팀은 학생이 실제로 정답에 가까워지고 있는지를 별도로 측정해, 선생님과 다르다는 이유만으로 억눌린 좋은 추론 구간을 찾아내 그 구간의 감점을 걸러내는 R2-OPD를 제안했다. DeepSeek-R1-Distill-Qwen-1.5B 실험에서 기존 OPD 대비 평균 정확도 2.51점, 정답 중 하나라도 맞힐 확률 4.46점을 끌어올렸다.

무엇을 했나

  1. OPD는 학생이 만든 답변을 선생님 모델의 확률분포와 비교해 토큰 단위로 점수를 매기는데, 이 점수가 실제 추론 진전을 제대로 반영하지 못하는 경우가 있다는 걸 확인했다
  2. 이를 해결하기 위해 학생이 중간 지점까지 쓴 내용만 가지고 정답을 맞힐 확률을 여러 번 시뮬레이션(Neval=8회)해서 '진짜 진전 점수(process reward)'를 별도로 계산했다
  3. 같은 방향으로 진전을 보이는 인접 구간들을 하나로 합쳐(sign-consistent merging) 노이즈를 줄인 뒤, 이 진전 점수 순위와 선생님 유사도 점수 순위가 어긋나는 구간을 찾아 해당 구간의 학습 신호를 걸러냈다(masking)
  4. DeepSeek-R1-Distill-Qwen-1.5B에서 JustRL을 선생님으로 쓴 실험 결과 기존 OPD 대비 평균 정확도(avg@4) 2.51점, 4개 중 하나라도 맞힐 확률(pass@4) 4.46점 향상했고, AIME 대회 수학 문제에서 특히 큰 개선을 보였다
  5. Qwen3-1.7B와 e3-1.7B 조합으로도 실험해 다른 모델 계열에서도 같은 방법이 통한다는 것을 확인했다
Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).
Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).
Table 1: OPD performance on DeepSeek-R1-Distill-Qwen-1.5B with JustRL as the teacher model.
MethodAIME 24AIME 25OlympiadAvg.
avg@4pass@4avg@4pass@4avg@4pass@4avg@4pass@4
Student22.5043.3323.3336.6743.1958.3129.6746.10
Teacher41.6756.6730.8343.4453.2868.4341.9256.18
OPD (1)28.3350.0022.5030.0046.8662.1032.5547.37
E-OPD (17)18.3336.6712.5023.3349.9164.5626.9141.52
TIP-OPD (36)17.5033.3311.6720.0048.2462.8325.8038.72
IW-OPD (34)20.8336.6717.5026.6743.5959.0927.3140.81
Uni-OPD (13)20.0043.3319.1726.6753.1669.9730.7846.66
R2-OPD (Ours)32.5056.6725.8336.6746.8662.1935.0651.83
Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment (J9) receives a smaller KL divergence under Equation (9) than segments that derive or recover the correct solution, respectively(J2 and J10).
Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment (J9) receives a smaller KL divergence under Equation (9) than segments that derive or recover the correct solution, respectively(J2 and J10).
Table 2: Transfer performance of R2-OPD on Qwen3-1.7B with e3-1.7B as the teacher model.
DatasetBaseOPDR2-OPD
avg@4pass@4avg@4pass@4avg@4pass@4
AIME 2424.1730.0022.5036.6725.0040.00
AIME 2518.7520.0027.5033.3325.8336.67
Olympiad50.1063.8753.8267.0954.3167.91
Avg.31.0137.9634.6145.7035.0448.19
Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.
Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.
Table 3: Effect of sign-consistent segment merging on avg@4 accuracy. Δ denotes the improvement of the full R2-OPD over its no-merging variant.
DatasetNo MergeR2-OPDΔ
AIME 2417.532.5015.0
AIME 2511.6725.8314.16
Olympiad47.846.86-0.96
Figure 4: Sensitivity of R2-OPD to the masking ratio q. A moderate ratio of q=30 provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.
Figure 4: Sensitivity of R2-OPD to the masking ratio q. A moderate ratio of q=30 provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.
Table 4: Optimization, process-reward estimation, and filtering settings used in all R2-OPD experiments.
SettingValue
Optimization and sequence settings
Training epochs1
OptimizerAdamW
Learning rate5×10−6
Global batch size64
Maximum prompt length1,024 tokens
Maximum response length7,168 tokens
KL support sizeStudent top-16 (H=16)
Process-reward estimation
Rollouts per evaluated boundaryNeval=8
Sampling temperature0.7
Top-k50
Top-p1.0
Maximum rollout length300 tokens
Segment filtering
Minimum sentences between boundariesSmin=3
Minimum merged segmentsnmin=3
Segment masking ratioq=30%
Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.
Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.
Table 5: Eligibility checks and conservative fallback actions used by R2-OPD.
ConditionAction
The string-level pre-check does not find gi in yi, or segmentation yields fewer than two segments.Skip process-reward rollouts and retain all token-level OPD supervision.
The response has fewer than max⁡(3,nmin) merged segments.Do not apply segment masking.
No strict adjacent pair is available after process-reward ranking.Do not apply segment masking.
All candidate segments have zero inconsistency score.Do not apply segment masking.
At least one candidate has a positive inconsistency score.Mask up to the response-specific budget; retain every other token.
Table 6: Complete lexicon of discourse markers used to propose reasoning-segment boundaries.
CategoryMatched markers
Reconsiderationwait, hold on, let me reconsider, hmm
Correctionactually
Verificationlet me check
Alternative reasoningalternatively

왜 중요한가

선생님 모델을 그대로 흉내 내게 하는 기존 지식 증류 방식은 학생이 독창적이지만 옳은 풀이 경로를 찾아도 벌점을 줄 수 있다는 구조적 약점을 드러낸 연구다. 추론 능력이 중요한 수학·코딩 AI를 더 저렴하게, 더 잘 훈련시키려는 회사나 연구자에게 실질적인 훈련 전략 개선안을 제시한다.

이 논문의 용어

  • On-Policy Distillation(OPD) · 학생 모델이 직접 만든 답변을 선생님 모델에게 채점받아 학습하는 지식 증류 방식
  • 역방향 KL 발산(reverse KL divergence) · 학생의 확률분포가 선생님의 확률분포와 얼마나 다른지 측정하는 수치로, 학습 신호로 쓰인다
  • 프로세스 리워드(process reward) · 중간 단계까지의 풀이만으로 정답을 맞힐 확률이 앞 단계 대비 얼마나 올랐는지를 나타내는 점수
  • sign-consistent merging(부호 일치 병합) · 진전 방향(플러스/마이너스)이 같은 인접 구간들을 하나로 묶어 노이즈를 줄이는 기법
  • avg@4 / pass@4 · avg@4는 4번 시도한 정답률의 평균, pass@4는 4번 중 한 번이라도 맞혔는지를 보는 지표

논문 원문 초록 (영문)

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

저자 · Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Chen Yang et al., arXiv:2608.19408, arxiv-nonexclusive