Dust Takes On Backprop

🔥 What happened Researchers unveiled Dust, a zeroth-order method that competes with backprop for pretraining transformer language models. Dust perturbs activations per token, treating each token as a virtual population member evaluated in parallel during one forward pass. 💡 Why it matters From 1M tokens up, Dust is extrapolated to be 10³–10⁴× more efficient than EGGROLL, a state-of-the-art ES method. Strikingly, larger models are more population-efficient: a 243M-parameter model beats a 120× smaller one at most population sizes. ⚡ Our take If compute keeps winning, backprop's differentiability crutch may become optional. Dust is still expensive, but the scaling trend is a warning shot to gradient orthodoxy.