Agents Fail at Production Reality

🔥 What happened Jennifer Williams' team dropped SWE-Serve: a benchmark of 53 tasks pulled from real SGLang production changes, testing agents on inference engineering. Best result across 11 models and 31 configs: 75% pass@1. 💡 Why it matters On 19 tasks with end-to-end coverage, roughly a third of patches that pass every other test get rejected by serving E2E tests — pass rate drops from 69.4% to 45.9%. Agents that ace local tests ship code that breaks in real serving environments. ⚡ Our take Benchmarking agents on local tests is theater. SWE-Serve proves the gap between "works on my machine" and "works in production" isn't a footnote — it's the whole story.