benchmark long-horizon-coding regression-testing claude-opus code-quality

Opus 5 reaches 24% on long-horizon coding benchmark

SlopCodeBench reveals multi-checkpoint progressive requirements expose model drift: Opus 5 accumulates defects steadily after initial wins, hitting regression on 93% of written code lines.

Summary

Standard benchmarks divulge entire problem upfront; SlopCodeBench forces iterative evolution with inherited state, exposing whether your LLM can maintain a codebase without human steering across multi-step requirements.

Why it matters

Implementation verdict

This replaces abstract code-quality metrics with concrete regression testing. Requires: running your own checkpoint-based evals if you care about real codebase maintenance. Ready now as a diagnostic tool, but Opus 5's 24% strict pass rate signals no model yet runs unattended on multi-step engineering work.

Sources

1.Opus 5 got a 24% on the small subset of the benchmark
2.THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality
3.each challenge in SlopCodeBench has multiple checkpoints - the model doesn't know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged
4.every dollar bought correctness. nobody bought enough of it.
5.For all 9 test runs, none of the models made it to the end of any challenge with everything passing, even on the problem marked as easy difficulty

Dev Signal

Get briefs like this in your inbox — free, every weekday.

100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.

Read the full issue →All briefs