Language Models Hop Between Parrot and Genius

🔥 What happened Researchers at GDsuite found that language models suddenly flip between shallow pattern-matching and genuine generalization during pre-training. On an arithmetic test, OLMo3-32B jumped from 81% to 0% and back again within just 0.04 trillion tokens. 💡 Why it matters This "mode-hopping" isn't noise: it persists under checkpoint averaging and can't be explained by single updates. The upside: an intermediate checkpoint at 4.5T tokens scored 36.3% on GPQA after SFT – far better than the later 4.9T checkpoint's 29.8%. ⚡ Our take Always picking the final checkpoint leaves performance on the table. The AI industry urgently needs tools to spot these generalization jumps early – otherwise we're training blind.