OpenAI's audit of SWE-Bench Pro found roughly 30% of public tasks were broken, exposing how evaluation flaws distort model capability rankings.
Summary
If you're comparing coding models or tracking progress via SWE-Bench scores, the baseline is corrupted. Flawed benchmarks mask real model gaps and waste engineering effort on false signals.
Why it matters
If you're comparing coding models or tracking progress via SWE-Bench scores, the baseline is corrupted. Flawed benchmarks mask real model gaps and waste engineering effort on false signals.
Implementation verdict
Don't rely on SWE-Bench Pro scores alone for model selection. Audit your own task definitions and reproduce benchmark tasks locally before shipping. The broader implication: synthetic benchmarks need continuous validation, not trust.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.