Generalization behavior is locally stable under a single Adam update on sampled OLMo3 data: suite probability changes are negligible across the tested checkpoints, including a learning rate of 10^-2 and a batch of 8K tokens. This analysis argues against explaining mode-hopping solely by one-step edge-of-stability oscillations. · CiteArk
Generalization behavior is locally stable under a single Adam update on sampled OLMo3 data: suite probability changes are negligible across the tested checkpoints, including a learning rate of 10^-2 and a batch of 8K tokens. This analysis argues against explaining mode-hopping solely by one-step edge-of-stability oscillations.
