Why On-Policy Distillation Works Even When the Teacher Is Wrong Half the Time

A close reading of Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement by Yi Ding and Ruqi Zhang.

On-policy distillation is supposed to be the well-behaved option. You let the student generate its own trajectories, you have a bigger teacher score every token, and instead of one sparse reward at the end of a rollout you get dense supervision everywhere. That is the pitch, and it is a good pitch.

Ding and Zhang measured how good that dense supervision actually is. On student-sampled trajectories, the sign of the teacher's token-level advantage disagrees with ground-truth correctness 30.6% of the time for a 4B teacher, 34.7% for a 30B-A3B teacher, and 50.6% for a 235B-A22B teacher. The biggest teacher is at coin-flip agreement with the truth.

Then they checked whether it mattered. Train on only the noisy trajectories, train on only the clean ones, or train normally, and the student converges to comparable performance in a comparable number of gradient steps.

That is the interesting part. Not that the supervision is noisy, but that the student is indifferent to it. If the signal can be half wrong and produce the same result, the signal is not what is doing the work.

What the advantage actually is

Start with the objective, because the whole result falls out of it. On-policy distillation minimises the reverse KL between student and teacher over trajectories the student sampled itself. Written as a policy gradient, each token carries an advantage:

Look at what that quantity is. It is positive when the teacher assigns more probability mass to the sampled token than the student did, and negative when it assigns less. It is a measure of disagreement in confidence. There is nothing in it that knows whether the trajectory ended in a correct answer.

Most of the time that is a reasonable proxy. A stronger model probably does put more mass on better continuations. But the proxy breaks in a specific and predictable place.

Consider the token inside \boxed{}. By the time the student emits it, it has already done the reasoning and committed to an answer, so ${\pi }_s$ for that token is close to 1. The ratio ${\pi }_t/{\pi }_s$ is then bounded above by roughly $1/{\pi }_s$, which is barely above 1. Whatever the teacher believes about the answer, the log ratio at a position where the student is already certain is pinned near zero or below it.

The paper measures exactly this for the 235B-A22B teacher: 97.8% of \boxed{} answer tokens get a negative advantage when the answer is correct, and 96.6% get one when it is wrong.

The teacher is a below-chance judge of its student's answers

Those two numbers are worth pushing on, and this next bit is my arithmetic rather than the paper's.

Treat the sign of the advantage on the answer token as a prediction: positive means "this trajectory is correct". Sensitivity is 2.2%. Specificity is 96.6%. Balanced accuracy is $(2.2+96.6)/2=49.4\text{\%}$, which is a shade under a coin flip.

Worse, the sign points the wrong way. A positive signal appears on 3.4% of incorrect trajectories and only 2.2% of correct ones. At the single position in the entire rollout that carries the answer, a vote of confidence from the teacher is very slightly evidence that the answer is wrong.

This also explains why the noise rate climbs with teacher size, which otherwise looks backwards. The teacher is scoring trajectories that are off-policy for it, because the student wrote them. A more capable teacher has a sharper and more idiosyncratic distribution, so its disagreement with the student's sampled tokens goes up, not down. Capability and agreement are different properties, and this objective only ever reads the second one.

Most tokens carry nothing at all

The other half of the diagnosis is about coverage rather than correctness. During on-policy distillation, 29.2% of tokens have exactly zero advantage and 51.7% have an advantage smaller than 1e-4 in magnitude. Half the supervision is numerically absent.

The paper also shows the converse: training only on the tokens where the student is most confident yields limited improvement, because those are precisely the tokens where the log ratio has no room to say anything. See Figure 3 in the paper, which is worth opening.

So the picture is a signal that is missing on half the tokens, is uninformative wherever the student is confident, and disagrees with the truth up to half the time where it does fire. And yet the training works.

Delete the teacher and see what is left

This is the experiment that turns an observation into a claim about mechanism. Replace the entire teacher-derived advantage with a single fixed negative number, applied to low-probability tokens, and the student improves about as much as it does under real on-policy distillation. Their Figure 4 is the load-bearing evidence.

A constant. No teacher forward pass, no distribution to compare against, no supervision of any kind. The only information left in the update is which of the student's own tokens were improbable, and that turns out to be most of what on-policy distillation was buying.

From there the method almost writes itself. OPSA, On-Policy Self-Adaptation, takes the lowest 20% of tokens by log probability and assigns each a negative advantage scaled by the entropy at that position:

$$r_i=\frac{2(H_i-H_{\min })}{H_{\max }-H_{\min }}-1$$ $$A^{\mathrm{dyn}}=A^{\mathrm{fix}}-\frac{1}{4}\delta r_i$$

With $A^{\mathrm{fix}}=-3/4$ and $\delta =1$ this reduces to

$$A^{\mathrm{dyn}}=-\frac{1}{2}-\frac{H_i-H_{\min }}{2(H_{\max }-H_{\min })}$$

which runs from $-1/2$ at the lowest-entropy positions to $-1$ at the highest. Every advantage is negative. The method never tells the model what it should have said. It only pushes down tail tokens, and it pushes hardest where the model was least sure.

One small thing that made the design click for me: $-3/4$ is exactly the midpoint of the $[-1,-1/2]$ range. The fixed-advantage ablation from the previous section is the entropy-averaged version of the adaptive rule, which is why it worked as well as it did.

Training uses DAPO-17k questions only, with no answers and no labels. There is no teacher, no verifiable reward and no reference answer anywhere in the loop.

The number in the abstract, and the number that calibrates it

The headline is that OPSA takes Qwen3-1.7B from 13.44 to 48.85 Avg@32 on AIME24, a gain of 35.41 points, and beats on-policy distillation by 16.77. Both are accurate against their Table 3, and worth pairing with a third: measured against GRPO, the strongest trained baseline in that table at 33.96, the margin is 14.89.

Here is the comparison I could not stop thinking about, and it is one the paper prints the ingredients for without drawing. Qwen3-1.7B ships with a thinking mode. Averaged across AIME24, AIME25 and HMMT25:

Qwen3-1.7B setting	Avg@32	Pass@32
base	9.62	31.11
base + OPSA	35.83	65.56
Thinking mode	35.24	71.11
Thinking mode + OPSA	40.71	72.22

35.83 against 35.24. Per benchmark the agreement is just as close: 48.85 against 46.56 on AIME24, 35.31 against 36.25 on AIME25, 23.33 against 22.92 on HMMT25.

A label-free training run, with no teacher and no rewards, lands a non-thinking model approximately where the same model's own thinking mode already was. Read one way that is deflationary. Read the other way it is the most interesting thing in the paper, because it says the capability was already in the weights and the entire job of training was changing which tokens get sampled.

The Pass@32 column is where the two stop agreeing: 65.56 against 71.11. And applied on top of thinking mode, OPSA adds only 5.47 Avg@32 and 1.11 Pass@32. The authors say the same thing plainly in their limitations, that OPSA "primarily operates by redistributing the policy's existing probability mass" and may not expand the exploration frontier. I would only add that this framing deserves to sit next to the headline rather than in an appendix.

The saving is memory, not step time

Their overhead table gives OPSA a 46.3s step against 61.2s for on-policy distillation and 186.2s for GRPO. A 24% cut over OPD is nice but it undersells the change.

My estimate, not theirs: on-policy distillation with the 235B-A22B teacher has to keep that teacher resident and run a forward pass through it every step. At BF16 that is about $235\times {10}^9\times 2\text{bytes}=470\text{GB}$ of weights before activations or KV cache, so roughly six 80 GB accelerators doing nothing but being the teacher. OPSA needs zero of them.

If you post-train on a small cluster, that is not a speedup, it is the difference between an experiment you can run and one you cannot.

Is it just longer outputs?

This was my first objection. OPSA generates 23,205 tokens per AIME24 problem against 4,457 for the base model and 15,286 for on-policy distillation, and inference time goes from 1.78s to 6.58s. Longer chains score better on competition maths, so some of the gain could be bought rather than learned.

They ran the control. Forcing GRPO and OPD out to comparable lengths by injecting "wait" makes both slightly worse, not better: GRPO drops from 33.96 to 32.81 at 23,261 tokens, OPD from 32.08 to 31.67 at 23,472 tokens, against OPSA's 48.85 at 23,205. Length alone does not buy the result.

My one reservation, and it is a reservation about the control rather than the finding: injecting "wait" forces length onto a model that was never trained to use it, which is not the same experiment as training a baseline to produce longer trajectories. It rules out the cheapest explanation, not every version of it.

What to take away

• The advantage in on-policy distillation is a confidence-disagreement signal, not a correctness signal. If you are debugging one, this is the assumption to check first, and it fails hardest exactly where the student is most confident.
• A bigger teacher is not automatically a better one here. Agreement with your student's sampled tokens is what the objective reads, and that gets worse with scale, not better.
• Before paying for a teacher, run the constant-advantage ablation. If a fixed negative number on the low-probability tail reproduces most of your gain, the teacher is a line item you can delete.
• Compare against the model's own thinking mode, not just its base mode. Otherwise you cannot tell recovered capability from new capability, and those have very different implications for what to do next.
• Watch Pass@k, not only Avg@k. Avg@k rising while Pass@k stays flat is the signature of redistribution rather than expansion.