SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

What they did

1. Motivation: RLVR grades an entire model response with a single score while OPD scores each token against a stronger teacher model but caps performance at the teacher's own quality; combining them seemed promising, but simply adding them with a fixed coefficient caused the model to lose diversity in its outputs early in training (entropy collapse).
2. Diagnosis: this happens because of two mismatches - a magnitude mismatch, where OPD's token-level scores occasionally spike far larger than the bounded RLVR score and drown it out, and a temporal mismatch, where sustained full-strength teacher-imitation keeps pulling the student toward the teacher and blocks it from exploring beyond the teacher's own ability.
3. Method: SAF applies a four-stage transformation only to the OPD signal - keeping just the top fraction of tokens by magnitude and zeroing the rest, compressing the surviving values through a bounded tanh function, ramping up OPD's strength during an early warm-up phase monitored by how much the student-teacher KL divergence has dropped, then gradually annealing the strength down afterward - before adding it to the unchanged RLVR score; each stage can be switched on or off independently.
4. Experiments: across Qwen3-1.7B, 4B, and 8B models on seven math reasoning and code generation benchmarks, SAF improved the aggregate score over fixed-coefficient fusion by 0.51-2.70 percentage points across all six model-domain settings, with more stable training and no entropy collapse.
5. Dynamics analysis: fixed-coefficient fusion tracked the teacher most closely (lowest student-teacher KL divergence) yet ended with the lowest final accuracy, while SAF maintained intermediate entropy and KL levels and finished with higher accuracy.

Findings

• Across seven math and code benchmarks with Qwen3-8B/4B/1.7B, SAF improved over fixed-coefficient GRPO+OPD fusion by 0.97%, 1.51%, and 1.85% on math for the three model sizes respectively, and by 1.67%, 2.70%, and 0.51% on code, reaching an average of 49.46% across six model-domain settings versus 1.54%, 2.71%, and 1.60% lower for fixed fusion, GRPO-only, and OPD-only respectively.
• In a 300-step ablation on Qwen3-4B math reasoning, magnitude control alone reached 44.35% (versus 44.38% for fixed fusion), adding warm-up alone reached 44.07%, enabling annealing raised it to 45.23%, and the full SAF configuration with threshold delta=0.2 reached 45.89%.
• Analysis of the first 10 training steps showed OPD advantage magnitudes were mostly near zero but with occasional spikes; among 180 inspected largest-magnitude tokens, every one exceeded its sequence's GRPO advantage magnitude, with OPD extremes reaching 20.3585 against a largest co-occurring GRPO magnitude of only 2.4749.
• Over 300 training steps, fixed-coefficient fusion rapidly dropped actor entropy from about 0.35 to 0.30 and drove student-teacher KL divergence to the lowest of four regimes tested, yet ended with the lowest accuracy on AIME-24 (about 0.57) and AIME-25 (about 0.51), while SAF kept entropy around 0.35-0.38 and finished higher on both (about 0.59 and 0.53 respectively).
• Layer-wise weight drift analysis showed fixed-coefficient fusion produced the largest absolute drift (peak about 0.03 by step 300, higher than GRPO-only) among regimes compared, while SAF's drift (peak about 0.04) was smaller than fixed fusion but still exceeded GRPO-only, indicating SAF curbs but does not eliminate excess parameter movement.

Where it can be used

• Designing post-training pipelines for language models that combine verifiable-reward RL with token-level teacher-distillation signals.
• Monitoring training runs for symptoms like rapid entropy drop or overly fast student-teacher KL convergence as diagnostic signals of instability.
• Considering SAF as a lightweight, drop-in replacement in an existing GRPO+OPD training loop without adding extra models or loss terms.

Limits and open work

• Experiments are limited to Qwen3-1.7B/4B/8B student models with a Qwen3-30B-A3B-Instruct-2507 teacher, and seven math and code generation benchmarks, so generalization to other model families or tasks is not established.
• SAF did not dominate every baseline at every scale - for example, GRPO-only or OPD-only outperformed SAF on Qwen3-8B's MBPP+ and on part of Qwen3-1.7B's code average.
• The layer-wise weight drift and update-matrix geometry analyses are explicitly described by the authors as supplementary, exploratory observations rather than mechanisms that drove SAF's design.
• Hyperparameters such as the KL threshold delta, top-k percentage, and tanh compression scale affected performance (e.g., raising delta from 0.2 to 0.3 lowered accuracy by 1.38 points), and their sensitivity across other tasks or model families is not fully explored.

Why it matters

As training pipelines increasingly combine verified-reward RL with teacher-imitation signals, this paper pinpoints exactly why naive combination fails and offers a lightweight fix that can be dropped into existing training loops. It gives practitioners building language model post-training pipelines a concrete diagnosis of the pitfalls in mixing these two signal types and a practical remedy.

Terms in this paper

• RLVR (reinforcement learning with verifiable rewards) · a reinforcement learning method that grades an entire model response with a single rule-checkable score, such as whether the final answer is correct
• OPD (on-policy distillation) · a training method that scores each token the student model generates against the probability a stronger teacher model would assign it
• entropy collapse · a training failure where the model stops trying diverse response paths and converges to repetitive, low-variety outputs
• GRPO · a widely used RLVR algorithm that samples multiple responses to the same prompt and normalizes their scores against each other into a group-relative advantage
• KL divergence · a measure of how different two probability distributions are - here, the student's and teacher's next-token predictions

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3