A small reasoning model can be wrong in a very specific way: it finds a plausible path, takes one bad turn, and then spends the next thousand tokens defending the turn. Standard on-policy distillation has a nasty weakness here. It asks the student to generate its own trajectory, then lets a stronger teacher score that trajectory. If the opening prefix is already broken, the teacher is grading a long mistake.
A new paper from Zhejiang University proposes a more surgical fix. Relay On-Policy Distillation, or Relay-OPD, lets the teacher briefly take control when the two models disagree about whether the reasoning should stop and rethink. The teacher then hands the sequence back. It is a small change to the training loop, but it attacks the part of distillation that wastes the most compute: continuing after the model has already lost the plot.
The failure happens before the bad answer
On-policy distillation is attractive because the student, not the teacher, supplies the prefixes used during training. That keeps training closer to the states the student will actually visit at inference time. The tradeoff is obvious once you watch long mathematical solutions. The student can commit to a wrong direction early, and every later token is conditioned on that error.
Relay-OPD calls this prefix failure. The paper's useful observation is that the failure leaves a live signal in the next-token distributions. On a bad prefix, the teacher often wants to produce a reflection or redirection token, words such as "But" or "Wait". The student instead prefers to continue the current line of reasoning. The teacher is preparing to change course while the student is pressing the accelerator.
The method turns that mismatch into a handoff trigger. At each position, it checks whether the teacher's top prediction is in a set of reflection tokens while those tokens are absent from the student's top-K choices. No verifier, reward model, or labelled error location is required. The system simply watches for a disagreement about whether the current reasoning deserves another look.
That is a better trigger than a generic probability gap. Two models can disagree strongly while still heading toward the same solution. Relay-OPD looks for a disagreement about direction, not just confidence.
When the trigger fires, the teacher generates a short leg of reasoning. The student then resumes from the repaired prefix. A relay budget limits how many teacher legs can occur and how long they can be, keeping the resulting trajectory close to the student's policy instead of quietly turning the whole rollout into teacher-generated data. The implementation uses one speculative-decoding engine: the student drafts, the teacher verifies, and accepted teacher-leg tokens follow the teacher distribution exactly.
The numbers are unusually clean
The experiments use a Qwen3-4B-Instruct-2507 teacher and two non-thinking students, Qwen3-0.6B and Qwen3-1.7B. The evaluation covers eight mathematics benchmarks: AIME 2024, AIME 2025, AIME 2026, MATH, AMC 2023, Olympiad problems, HMMT February 2026, and HMMT November 2025.
The 1.7B student is the more interesting result. Standard OPD reaches a mean accuracy of 41.23. FastOPD improves that to 45.47. Relay-OPD reaches 46.96, a 5.73 percentage-point gain over OPD and 1.49 points over FastOPD. It wins or places second on every benchmark in the table. The biggest gains over OPD appear on AIME 2024, AIME 2025, AIME 2026, and AMC 2023, where the margins are 6.88, 7.29, 7.19, and 6.80 points.
The 0.6B student also improves, although its absolute performance is much lower. Relay-OPD posts a mean of 31.04 against 28.03 for OPD and 30.42 for FastOPD. On this smaller model, the training trajectory averages 2,490 response tokens, compared with 6,900 for OPD. That is a 63.9 percent reduction. For the 1.7B student, the corresponding lengths are 2,296 and 4,658 tokens, a 50.7 percent cut.
The paper also reports a revealing intervention test before the full method. Replacing only the teacher's reflection token at trigger positions means teacher tokens make up just 0.35 percent of the generated sequence. Accuracy rises from 27.73 to 34.96, a 7.23-point jump. That suggests the important resource is not a long teacher monologue. It is the timely decision to stop following the wrong branch.
Timing matters more than brute force. In one experiment, moving interventions away from the earliest triggers drops accuracy from 41.99 to 33.98 and then 29.49 as the delay grows. Once a bad prefix gets longer, even the teacher becomes more constrained by the context it has been handed. A late rescue is still reading the same damaged map.
What this does not prove
The result is strong, but the scope is narrow. Every reported experiment uses Qwen3 teacher-student pairs and mathematical reasoning. The authors have not shown that the same reflection-token list works for code generation, tool use, or multimodal agents. A coding model may signal a failed plan with a shell command, a test failure, or a change in file-selection strategy rather than a word like "Wait".
There is also a capability assumption hiding in the trigger. The teacher must be better than the student at recognizing when the reasoning should change direction. If the teacher and student are close in capability, or if they share the same blind spot, the signal becomes weak. The paper says the relay budget was tuned on the 1.7B student and reused for the 0.6B model. Its sensitivity study shows reasonable stability, but a different model family may need different trigger tokens and budgets.
The comparison set matters too. Relay-OPD is designed to improve on-policy distillation, not to replace every form of reinforcement learning or supervised fine-tuning. Its advantage comes from preserving the student's visited states while selectively repairing the worst prefixes. That makes it especially appealing when teacher inference is expensive and long student rollouts are full of low-value tokens.
My take is that the paper's best idea is smaller than its headline result. The useful lesson is not "add a teacher to every generation." It is "find the moment when the student stops reconsidering." A teacher that intervenes for a fraction of a percent of tokens can change the rest of the trajectory, provided it acts before the error hardens into context.
That is a practical design rule for future distillation systems. Do not only ask which tokens the teacher prefers. Ask whether the teacher and student still agree on the direction of the reasoning. If they do not, pass the baton early.
For the full method, ablations, token lists, and implementation details, see the Relay-OPD project page, the arXiv paper, the Hugging Face paper page, and the released GitHub code.