Training AI Across 9,000 Miles of Clusters

🔥 What happened A team trained a 32B-parameter sparse mixture-of-experts model across two clusters 9,000 miles apart, using different GPUs (GB300 and B300) and a 25 Gbps link. They swapped all-reduce for butterfly mixing, where eight learners exchange model fragments in pairs. 💡 Why it matters At equal compute, the distributed run landed just 1% worse in bits-per-byte than standard training. That cracks the assumption that frontier training needs one monolithic cluster. Goldman Sachs expects only 50–60% of US data center capacity scheduled for the next 1–2 years to come online on time — the rest is stranded compute. ⚡ Our take The real message is uncomfortable for anyone betting on one giant cluster: the bottleneck isn't chip count, it's the ability to fuse scattered islands into a single training run. Whoever solves that suddenly taps a far larger compute pool than the competition.