Removing On-Policy Distillation: When It Helps, When It Hurts, and Why

On-policy distillation provides dense, token-by-token monitoring of training thought models; however, it remains unclear under which conditions this signal is beneficial and under which it is harmful. What teacher model should be used, and in the case of self-reflection, what specific context should serve as a guiding signal? Does the best choice vary from one token to the next? Currently, answering these questions often requires expensive training with aggregated performance metrics that hide the power at the level of individual tokens. We present a free diagnostic framework that works with the highest resolution: per token, per question, and per teacher. We find the optimal gradient for each node which is defined as the parameter update that maximizes the student's success probability. We then develop a scaling target extraction algorithm to estimate this gradient well, even for long chains of intermediate thoughts. The gradient alignment score, defined as the cosine similarity between this ideal gradient and any given distillation gradient, quantifies the degree to which a given concentration reaches an ideal signal. Across a range of self-reflection settings and external teacher models, we see that the distillation approach shows a much higher and better agreement in the wrong extraction than in the right one, when the student is already doing well and the teacher's signal is often noisy. Additionally, we find that the optimal context for immersion distillation depends jointly on the learner's model capacity and the target task, and no single universally effective configuration emerges. This finding motivates the use of each function, analyzing each token for distillation analysis.