Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation

For Qwen3-4B-Math→Qwen3-4B mathematical distillation, Table 1 reports γOPD as the best method on both average accuracy (69.49) and average pass rate (82.41), exceeding the teacher averages (68.10 and 79.81) and the strongest OPD-baseline averages.

Without RBM in Qwen3-4B-Math→Qwen3-1.7B training, γ=1 produces much larger gradient norms, rapid policy-entropy collapse, and inferior AIME24 validation accuracy. Local OPD (γ=0) and discounted variants improve steadily, with γ=0.99 reported as best and better than γ=0.9.

Profiling on the main 4B student/teacher and 32-GPU setup reports total additional γOPD work of 610.32 ms/step and 0.375 MB, equal to 0.1064% time and 0.00046% memory relative to a vanilla OPD update.

Under joint math-and-code multi-teacher distillation into Qwen3-4B, γOPD records TotalAvg 66.00, the highest table value; it leads HumanEval+ and LiveCodeBench among code tasks and exceeds the teacher on the paper's average math and code comparisons, but not every individual benchmark or metric.

For the displayed incorrect response, local A_t^(0) penalizes tokens only weakly related to the actual reasoning error, whereas A_t^(0.99) concentrates stronger negative credit on the erroneous formula derivation without excessively penalizing the final end-of-sequence token.

In vanilla-distillation training curves, the paper reports that γOPD maintains the highest verifiable reward, stable absolute OPD advantage, and the smallest gradient-norm fluctuations; its response length becomes shorter and more stable than most baselines. TOPD is the exception on length because it truncates the distillation signal, and its verifiable reward remains mostly below zero.

For Qwen3-4B-Math→Qwen3-1.7B mathematical distillation, every trained student remains below the teacher on the two reported averages, while γOPD gives the best trained-student average accuracy (56.51) and average pass rate (71.22).

For the displayed correct response, local A_t^(0) gives sparse supervision and noticeable negative credit to only a few tokens, whereas normalized A_t^(0.99) propagates later information, smooths the signal, assigns stronger positive credit to the key mathematical derivation, and mildly penalizes redundant text.

On AIME24 and AIME25, temporal discounting alone raises average accuracy by 2.04 points over vanilla OPD; naive reward mixing adds only 0.05 point beyond discounting, while the full discounting-plus-mixing-plus-normalization configuration reaches 57.56 average accuracy, a 3.91-point gain. Mixing plus normalization without discounting gains 2.00 points, supporting complementary contributions.