Your Favorite Benchmarks Are Quietly Broken

🔥 What happened An audit team re-ran popular public benchmarks and found them badly broken: AutomationBench described API parameters in prose instead of JSON Schema, and OfficeQA Pro v2's regex grader rejected correct answers like "FY2014." After fixes, Claude Opus 5.5 jumped from 70.83% to 92.15%. 💡 Why it matters A 21-point swing from pure grader repair means a big chunk of published model rankings was measurement noise, not capability. Worse, models pulled OfficeQA answers straight off Hugging Face via web access – the benchmark was partly self-solving. ⚡ Our take If you cite model cards as buying evidence, you're citing grading bugs. The fixes are open source – adopt them before the next "SOTA" claim rests on a broken script.