More Tasks Won't Save Your Agent Leaderboard

🔥 What happened Stanford-led researchers ran a Bayesian variance decomposition across 22 agent benchmarks (Holistic Agent Leaderboard, Harbor Index). Verdict: leaderboard ranks often measure the scaffold, not the model. 💡 Why it matters Underlying-model reliability drops as low as 0.148, while fixed model-scaffold systems hit 0.935–0.994. Adding infinitely many similar tasks lifts model-ranking reliability by at most 0.097 when scaffold coverage dominates. Pooling diverse benchmarks jumps reliability from 0.44 to 0.75 at the same task budget — and cuts projected cost by up to 83%. ⚡ Our take Treating agent leaderboards as model truth is self-deception. Stop stacking tasks; start diversifying scaffolds.