Figure 1: W2S-OPD improves Qwen3-8B using 4B models as teachers. Accuracy is averaged over 4 math reasoning and 3 code generation benchmarks. (a) With a post-RL Qwen3-4B expert, W2S-OPD beats OPD. (b) From 2 off-the-shelf base models (Qwen3-4B and 0.6B), both weaker than the student and used without training, W2S-OPD still lifts the student above itself and both sources.
Table 1: The three instantiations of the contrast pair in W2S-OPD. All models are from Qwen3 series.
Setting
Positive model m+
Negative model m−
Anchor zbase
Pre-RL / Post-RL
4B-RL (post-RL expert)
4B (pre-RL init)
8B (base)
Smaller / Larger
4B (base)
0.6B (base)
8B (base)
Correct / Wrong Hints
4B w/ correct hint
4B w/ wrong hint
8B (base)
Figure 2: Overview of W2S-OPD. W2S-OPD synthesizes a proxy teacher and distills it into a student. A positive and a negative model form a contrast pair whose logit difference isolates a capability direction. The student πS, initialized from the same base model, generates on-policy rollouts and minimizes the per-token reverse KL toward this proxy teacher. The contrast pair can be instantiated as a post-RL expert against its pre-RL initialization, a larger base model against a smaller one, or a single model conditioned on correct against wrong hints.
Table 2: Results for Pre-RL / Post-RL contrast setting. W2S-OPD beats OPD and even surpasses the 4B-expert on math. Improv. reports the absolute gain over OPD.
Method
Math Reasoning
Code Generation
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg.
HumanEval+
MBPP+
LCB
Avg.
Neg. Model (4B)
21.6
19.9
10.0
8.2
14.9
79.9
63.6
20.9
54.8
Pos. Model (4B-RL)
62.0
57.9
34.8
40.4
48.8
86.3
71.0
27.1
61.5
Student Base (8B)
26.1
21.1
11.6
9.0
17.0
81.2
70.0
21.7
57.6
Single-Teacher Distillation
SFT
60.2
54.4
30.7
37.6
45.7
82.5
71.8
24.8
59.7
OPD
62.1
54.8
30.6
38.3
46.5
83.1
71.2
22.0
58.7
W2S-OPD
68.9
60.1
35.1
43.1
51.8
84.8
72.4
26.8
60.9
Improv.
+6.8
+5.3
+4.5
+4.8
+5.3
+1.7
+1.2
+5.0
+2.2
Multi-Teacher Distillation
SFT
63.5
54.6
29.6
39.4
46.8
84.0
71.4
25.7
60.4
OPD
64.3
54.9
29.6
37.4
46.5
83.4
69.5
25.6
59.5
W2S-OPD
67.4
61.0
34.7
45.2
52.1
85.2
71.0
27.1
61.1
Improv.
+3.1
+6.1
+5.1
+7.8
+5.6
+1.8
+1.5
+1.5
+1.6
Figure 3: Performance on the math and code benchmarks over training steps. W2S-OPD improves faster and outperforms OPD.
Table 3: Results for contrastive hints setting. W2S-OPD improves the 8B student above its own base with only a single weaker and smaller model. Improv. reports the absolute gain of W2S-OPD over the student base model.
Method
Math Reasoning
Code Generation
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg.
HumanEval+
MBPP+
LCB
Avg.
Pos./Neg. Model (4B)
21.6
19.9
10.0
8.2
14.9
79.9
63.6
20.9
54.8
Student Base (8B)
26.1
21.1
11.6
9.0
17.0
81.2
70.0
21.7
57.6
W2S-OPD
27.7
23.3
12.0
10.7
18.4
82.5
70.8
22.7
58.7
Improv.
+1.6
+2.2
+0.4
+1.7
+1.4
+1.3
+0.8
+1.0
+1.1
Figure 4: Performance on the math and code benchmarks with different α. OPD is included for reference, denoted by the gray line. W2S-OPD outperforms OPD over a wide range of α.
Table 4: Results for Smaller and Larger contrast setting. W2S-OPD improves the 8B student above its own base even though both source models are weaker than it. Improv. reports the absolute gain of W2S-OPD over the student base model.
Method
Math Reasoning
Code Generation
AIME24
AIME25
HMMT25 (Feb.)
HMMT25 (Nov.)
Avg.
HumanEval+
MBPP+
LCB
Avg.
Neg. Model (0.6B)
1.6
2.3
0.1
2.7
1.7
25.6
26.3
4.6
18.8
Pos. Model (4B)
21.6
19.9
10.0
8.2
14.9
79.9
63.6
20.9
54.8
Student Base (8B)
26.1
21.1
11.6
9.0
17.0
81.2
70.0
21.7
57.6
W2S-OPD
33.1
25.9
13.0
20.1
23.0
81.7
71.1
23.7
58.8
Improv.
+7.0
+4.8
+1.4
+11.1
+6.0
+0.5
+1.1
+2.0
+1.2
Table 5: Results for OOD generalization on GPQA-Diamond and IFBench. Both distillation methods are trained only on the math task, W2S-OPD transfers out of domain and improves general ability, whereas OPD can degrade it below the base. Improv. indicates the absolute gain over OPD.
Method
GPQA-Diamond
IFBench
Positive Model
51.8
26.0
Student Base
38.9
26.3
OPD
54.4
25.9
W2S-OPD
56.5
27.0
Improv.
+2.1
+1.1
Table 6: Distribution of the top-1% highest-Δ tokens over the eight Schoenfeld episodes. The first row gives the distribution of each episodes of all tokens.
Read
Analyze
Plan
Implement
Explore
Verify
Monitor
Answer
All tokens
10.3
33.2
8.6
40.7
0.5
2.4
0.5
3.8
Pre-RL / Post-RL
9.6
40.5
15.7
24.0
1.2
4.3
1.8
2.9
Smaller / Larger
6.5
44.4
12.6
27.7
0.8
4.2
0.9
2.9
Correct / Wrong Hints
5.2
29.8
20.2
19.6
1.1
5.4
2.3
16.4
Table 7: Training hyperparameters of W2S-OPD. Paired entries denote math / code.
Hyperparameter
Value
Algorithm
On-Policy Distillation
Student Model
Qwen3-8B
Positive / Negative
Qwen3-4B-RL / Qwen3-4B
Amplification Coeff. (α)
1.0 / 0.75
Teacher Top-K
32
Max Prompt Length
2048
Max Response Length
16384
Learning Rate
2e-6
LR Scheduler
Constant
Sampling Temp. / Top-p
1.0 / 1.0
Batch Size
64
Training Steps
100
Table 9: On-policy rollout prompts used during OPD training.
Domain
Prompt
Math
{problem}\n Please reason step by step, and put your final answer within \boxed{}.
Code
{problem}\n Write Python code to solve the problem. Present the code in \n‘‘‘python\n Your code\n‘‘‘\n at the end. You need to think first then write the Python code.
Table 10: Average wall-clock time per training step (s) in the pre-RL / post-RL setting; W2S-OPD adds only 20% over OPD.
Method
Training time per step (s)
OPD
868
W2S-OPD
1043
Table 11: Average math reasoning accuracy of W2S-OPD with base-model contrast pairs of different capability gaps. The student is Qwen3-8B.
On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's
