Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Evaluated the complete 1B model family across all four post-training stages (Base, SFT, DPO, Instruct) on the paper's 240 Dolmino document benchmark (107,555 tokens); the 7B and 13B model suites were omitted due to cumulative network bandwidth and runtime budget limits (>220 GB total checkpoint download size across remaining 8 models).

Unresolved metric interpretation: fig8_roc_auc_7b_base: 7B checkpoint evaluation is constrained by total runtime budget and bandwidth limits; 12 checkpoints across 7B/13B total 221GB which exceeds execution budget.; fig8_roc_auc_7b_sft: 7B checkpoint evaluation is constrained by total runtime budget and bandwidth limits; 12 checkpoints across 7B/13B total 221GB which exceeds execution budget.; fig8_roc_auc_7b_dpo: 7B checkpoint evaluation is constrained by total runtime budget and bandwidth limits; 12 checkpoints across 7B/13B total 221GB which exceeds execution budget.; fig8_roc_auc_7b_instruct: 7B checkpoint evaluation is constrained by total runtime budget and bandwidth limits; 12 checkpoints across 7B/13B total 221GB which exceeds execution budget.; fig8_roc_auc_13b_base: 13B checkpoint evaluation is constrained by total runtime budget, network bandwidth (153GB for 13B suite), and single GPU VRAM limits.; fig8_roc_auc_13b_sft: 13B checkpoint evaluation is constrained by total runtime budget, network bandwidth (153GB for 13B suite), and single GPU VRAM limits.; fig8_roc_auc_13b_dpo: 13B checkpoint evaluation is constrained by total runtime budget, network bandwidth (153GB for 13B suite), and single GPU VRAM limits.; fig8_roc_auc_13b_instruct: 13B checkpoint evaluation is constrained by total runtime budget, network bandwidth (153GB for 13B suite), and single GPU VRAM limits.

Unresolved measurements: fig8_roc_auc_7b_base: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_7b_base.json; fig8_roc_auc_7b_sft: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_7b_sft.json; fig8_roc_auc_7b_dpo: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_7b_dpo.json; fig8_roc_auc_7b_instruct: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_7b_instruct.json; fig8_roc_auc_13b_base: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_13b_base.json; fig8_roc_auc_13b_sft: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_13b_sft.json; fig8_roc_auc_13b_dpo: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_13b_dpo.json; fig8_roc_auc_13b_instruct: The submitted result did not promote parser evidence at results/fig8_fig8_roc_auc_13b_instruct.json

Evaluated only OLMo-2 1B checkpoints; all 8 measurements for OLMo-2 7B and 13B models across Base, SFT, DPO, and Instruct stages were omitted due to checkpoint download sizes (>220 GB total) and budget limits.

The 240 Dolmino evaluation documents were reconstructed by sampling and token adjustment from Dolmino Mix 1124 rather than using the exact fixed author test split/seed.

The claim posits that teacher predictive entropy separates procedural from knowledge-intensive domains with high ROC AUC across training stages (Base, SFT, DPO, Instruct) and model sizes (1B, 7B, 13B). However, the executed reproduction only evaluated the 1B model checkpoints due to download size and runtime budget constraints. The 8 measurements for 7B and 13B models across all stages were not evaluated, leaving the claim coverage partial. Furthermore, the 240 documents used were constructed via synthetic token adjustment/sampling rather than the exact fixed evaluation subset from the paper, introducing minor protocol divergence. While the 1B measurements were successfully executed and reproduced high ROC AUC values (~0.79-0.81), the entire claim remains inconclusive due to missing multi-model coverage. Some original claim measurements remain without evidence; available measurements are assessed individually.