AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
Weak-to-Strong On-Policy Distillation
arXiv:2607.262462026-07-27
Distilling a stronger student AI using only weaker models than itself
On-policy distillation (OPD) has always required a teacher model at least as capable as the student, which fails once a model is already at the frontier. This paper's W2S-OPD extracts a 'capability direction' from the difference between two models weaker than the student and adds it onto the student's own base model, and it beats standard OPD on math and code benchmarks, sometimes even surpassing the weak teacher itself. The author is Fangxu Yu, and code is planned to be released.
METAL LAB explanatory visual
W2S-OPD structure: extracting a capability direction from weak pairs and injecting it into the student
Evidence statusMeasured results reported
Contrast pair (m+, m-)Take logits from two models weaker than the student, e.g., post-RL vs pre-RL, larger vs smaller base model, or correct-hint vs wrong-hint condition.
Extract capability directionSubtracting m- logits from m+ logits cancels what they share and leaves only the part where m+ is better.
Synthesize proxy teacherThis direction, scaled by α, is added onto the student's own base model logits, producing a teacher close to the student but with an improved capability.
Distill on student's own rolloutsThe student generates its own answers and is trained to minimize reverse KL divergence against the proxy teacher's token distribution.
Evaluate resultsPerformance is measured against standard OPD on math and code benchmarks plus out-of-domain tasks (GPQA-Diamond, IFBench).
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
Problem: existing on-policy distillation needs a teacher at least as strong as the student, so once a model is already near the frontier there is no stronger teacher left to distill from.
Method: W2S-OPD takes the logit (per-token score) difference between a relatively strong model (m+) and a weaker one (m-), which cancels out what they share and isolates a 'capability direction,' then adds this direction onto the student's own base model to synthesize a proxy teacher.
This contrast pair can be built three ways: a post-RL expert versus its pre-RL initialization, a larger base model versus a smaller one, or a single model conditioned on a correct hint versus a wrong hint.
Using only Qwen3-4B-scale or smaller models as ingredients for a Qwen3-8B student, W2S-OPD outperformed standard OPD on average across math and code benchmarks, and with a post-RL 4B expert the student even surpassed that expert itself on math.
Even with two weak base models used without any extra training, the student improved above both its own base and the source models, and on out-of-domain tasks (GPQA-Diamond, IFBench) its general ability was preserved or improved.
Figure 1: W2S-OPD improves Qwen3-8B using 4B models as teachers. Accuracy is averaged over 4 math reasoning and 3 code generation benchmarks. (a) With a post-RL Qwen3-4B expert, W2S-OPD beats OPD. (b) From 2 off-the-shelf base models (Qwen3-4B and 0.6B), both weaker than the student and used without training, W2S-OPD still lifts the student above itself and both sources.
Table 1: The three instantiations of the contrast pair in W2S-OPD. All models are from Qwen3 series.
Setting
Positive model m+
Negative model m−
Anchor zbase
Pre-RL / Post-RL
4B-RL (post-RL expert)
4B (pre-RL init)
8B (base)
Smaller / Larger
4B (base)
0.6B (base)
8B (base)
Correct / Wrong Hints
4B w/ correct hint
4B w/ wrong hint
8B (base)
Figure 2: Overview of W2S-OPD. W2S-OPD synthesizes a proxy teacher and distills it into a student. A positive and a negative model form a contrast pair whose logit difference isolates a capability direction. The student πS, initialized from the same base model, generates on-policy rollouts and minimizes the per-token reverse KL toward this proxy teacher. The contrast pair can be instantiated as a post-RL expert against its pre-RL initialization, a larger base model against a smaller one, or a single model conditioned on correct against wrong hints.
Table 2: Results for Pre-RL / Post-RL contrast setting. W2S-OPD beats OPD and even surpasses the 4B-expert on math. Improv. reports the absolute gain over OPD.
Method
Math Reasoning
Code Generation
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg.
HumanEval+
MBPP+
LCB
Avg.
Neg. Model (4B)
21.6
19.9
10.0
8.2
14.9
79.9
63.6
20.9
54.8
Pos. Model (4B-RL)
62.0
57.9
34.8
40.4
48.8
86.3
71.0
27.1
61.5
Student Base (8B)
26.1
21.1
11.6
9.0
17.0
81.2
70.0
21.7
57.6
Single-Teacher Distillation
SFT
60.2
54.4
30.7
37.6
45.7
82.5
71.8
24.8
59.7
OPD
62.1
54.8
30.6
38.3
46.5
83.1
71.2
22.0
58.7
W2S-OPD
68.9
60.1
35.1
43.1
51.8
84.8
72.4
26.8
60.9
Improv.
+6.8
+5.3
+4.5
+4.8
+5.3
+1.7
+1.2
+5.0
+2.2
Multi-Teacher Distillation
SFT
63.5
54.6
29.6
39.4
46.8
84.0
71.4
25.7
60.4
OPD
64.3
54.9
29.6
37.4
46.5
83.4
69.5
25.6
59.5
W2S-OPD
67.4
61.0
34.7
45.2
52.1
85.2
71.0
27.1
61.1
Improv.
+3.1
+6.1
+5.1
+7.8
+5.6
+1.8
+1.5
+1.5
+1.6
Figure 3: Performance on the math and code benchmarks over training steps. W2S-OPD improves faster and outperforms OPD.
Table 3: Results for contrastive hints setting. W2S-OPD improves the 8B student above its own base with only a single weaker and smaller model. Improv. reports the absolute gain of W2S-OPD over the student base model.
Method
Math Reasoning
Code Generation
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg.
HumanEval+
MBPP+
LCB
Avg.
Pos./Neg. Model (4B)
21.6
19.9
10.0
8.2
14.9
79.9
63.6
20.9
54.8
Student Base (8B)
26.1
21.1
11.6
9.0
17.0
81.2
70.0
21.7
57.6
W2S-OPD
27.7
23.3
12.0
10.7
18.4
82.5
70.8
22.7
58.7
Improv.
+1.6
+2.2
+0.4
+1.7
+1.4
+1.3
+0.8
+1.0
+1.1
Figure 4: Performance on the math and code benchmarks with different α. OPD is included for reference, denoted by the gray line. W2S-OPD outperforms OPD over a wide range of α.
Table 4: Results for Smaller and Larger contrast setting. W2S-OPD improves the 8B student above its own base even though both source models are weaker than it. Improv. reports the absolute gain of W2S-OPD over the student base model.
Method
Math Reasoning
Code Generation
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg.
HumanEval+
MBPP+
LCB
Avg.
Neg. Model (0.6B)
1.6
2.3
0.1
2.7
1.7
25.6
26.3
4.6
18.8
Pos. Model (4B)
21.6
19.9
10.0
8.2
14.9
79.9
63.6
20.9
54.8
Student Base (8B)
26.1
21.1
11.6
9.0
17.0
81.2
70.0
21.7
57.6
W2S-OPD
33.1
25.9
13.0
20.1
23.0
81.7
71.1
23.7
58.8
Improv.
+7.0
+4.8
+1.4
+11.1
+6.0
+0.5
+1.1
+2.0
+1.2
Table 5: Results for OOD generalization on GPQA-Diamond and IFBench. Both distillation methods are trained only on the math task, W2S-OPD transfers out of domain and improves general ability, whereas OPD can degrade it below the base. Improv. indicates the absolute gain over OPD.
Method
GPQA-Diamond
IFBench
Positive Model
51.8
26.0
Student Base
38.9
26.3
OPD
54.4
25.9
W2S-OPD
56.5
27.0
Improv.
+2.1
+1.1
Table 6: Distribution of the top-1% highest-Δ tokens over the eight Schoenfeld episodes. The first row gives the distribution of each episodes of all tokens.
Read
Analyze
Plan
Implement
Explore
Verify
Monitor
Answer
All tokens
10.3
33.2
8.6
40.7
0.5
2.4
0.5
3.8
Pre-RL / Post-RL
9.6
40.5
15.7
24.0
1.2
4.3
1.8
2.9
Smaller / Larger
6.5
44.4
12.6
27.7
0.8
4.2
0.9
2.9
Correct / Wrong Hints
5.2
29.8
20.2
19.6
1.1
5.4
2.3
16.4
Table 7: Training hyperparameters of W2S-OPD. Paired entries denote math / code.
Hyperparameter
Value
Algorithm
On-Policy Distillation
Student Model
Qwen3-8B
Positive / Negative
Qwen3-4B-RL / Qwen3-4B
Amplification Coeff. (α)
1.0 / 0.75
Teacher Top-K
32
Max Prompt Length
2048
Max Response Length
16384
Learning Rate
2e-6
LR Scheduler
Constant
Sampling Temp. / Top-p
1.0 / 1.0
Batch Size
64
Training Steps
100
Table 9: On-policy rollout prompts used during OPD training.
Domain
Prompt
Math
{problem}\n Please reason step by step, and put your final answer within \boxed{}.
Code
{problem}\n Write Python code to solve the problem. Present the code in \n‘‘‘python\n Your code\n‘‘‘\n at the end. You need to think first then write the Python code.
Table 10: Average wall-clock time per training step (s) in the pre-RL / post-RL setting; W2S-OPD adds only 20% over OPD.
Method
Training time per step (s)
OPD
868
W2S-OPD
1043
Table 11: Average math reasoning accuracy of W2S-OPD with base-model contrast pairs of different capability gaps. The student is Qwen3-8B.
Contrast Pair
4B−0.6B
4B−1.7B
W2S-OPD
23.0
21.7
Improv. over Student
+6.0
+4.7
Findings
In the pre-RL/post-RL setting, W2S-OPD outperformed OPD by 11.4% relative on math under single-teacher distillation and 12.0% relative under multi-teacher distillation, and on math the student even surpassed the 4B domain expert itself.
Using only two off-the-shelf base models of different sizes (Qwen3-4B, 0.6B) without any additional training, W2S-OPD still lifted the student by an absolute 6.0% on math and 1.2% on code on average.
Using contrastive hints from a single 4B model, W2S-OPD improved the student by an absolute 1.4% on math and 1.1% on code on average.
On out-of-domain evaluation, W2S-OPD raised GPQA-Diamond from 38.9 to 56.5, an absolute 2.1% gain over OPD, while on IFBench OPD degraded below the base model but W2S-OPD stayed 1.1% above OPD.
Measuring wall-clock time per training step, W2S-OPD added only about 20% overhead over OPD despite forwarding three frozen models instead of one.
Where it can be used
Designing post-training pipelines that combine multiple existing smaller models to keep improving a frontier-scale student model.
Transferring the domain skill of a small, RL-specialized expert model into a larger model at lower training cost.
Data-efficient distillation experiments that extract a learning signal from a single model by only changing its context, such as correct versus wrong hints.
Limits and open work
All experiments used the Qwen3 family (8B student, 4B/0.6B/1.7B contrast pairs), so it is unverified whether the same effect holds for other model families or scales.
The amplification coefficient α is sensitive: too small yields a weak signal and too large distorts the teacher, so tuning is required.
Evaluation is limited to math, code, and two out-of-domain tasks (GPQA-Diamond, IFBench), so effectiveness in other real-world domains remains unverified.
The authors themselves note that scaling the composition of multiple contrast pairs (setting multiple αk values) needs further exploration.
How far weak signals can push a strong student before the approach saturates has not yet been established.
Why it matters
Frontier models can hit a wall because there's no stronger teacher to learn from, and this method shows a path to keep improving them using only the differences between existing, cheaper, weaker models. That could ease the ever-rising training cost problem in pushing AI capability further.
Terms in this paper
On-Policy Distillation (OPD) · A training method where a teacher model scores the student's own self-generated answers token by token to guide it
logit · The raw score a model assigns to each candidate next word before turning it into probabilities
reverse KL divergence · A measure of how different the student's probability distribution is from the teacher's, which training minimizes
GRPO · A reinforcement learning technique used to specialize a model in a domain like math or code
amplification coefficient α · A value controlling how strongly the capability direction is injected into the proxy teacher
Original abstract (English)
On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's
