One in three benchmark tasks is broken

🔥 What happened A Stanford team audited Terminal-Bench 2.0 and found defects in roughly 30% of its tasks. The repaired 2.1 release lifts accuracy by over 10 percentage points and shifts leaderboard positions by up to three ranks. 💡 Why it matters Coding agents are economically consequential, yet even carefully vetted benchmarks rot as models improve. Extending the time limit alone closes a 20pp harness gap on GPT 5.4 — resource budgets are part of the measurement spec, not an afterthought. ⚡ Our take Treating leaderboards as ground truth means measuring benchmark debt, not capability. Continuous validation must become the default — otherwise we optimize against broken targets.