agent-benchmarking stateful-workflows reliability-testing mcp-tools evaluation

ThinkingBox grades agents on backend state, not tool calls

Measure agent reliability by executing tasks 20 times against isolated backends and checking terminal state changes, not parsing trajectories—reveals 67% of clean-looking failures have wrong field values or missing side effects.

Summary

Single-attempt benchmarks hide consistency gaps: Claude Opus 5.5 retains only 71% of its pass@1 score across 20 repeats. You need to know whether your agent will execute stateful workflows correctly every time, not just once.

Why it matters

Implementation verdict

Replaces pass@1-only leaderboard thinking with pass@20 and observed 20/20 metrics. Requires isolated MCP tool sessions and executable state checks on real backend objects. Available now via Hugging Face—worth running on your own workflows before production deployment to surface silent failures current evals miss.

Sources

1.grades agents on terminal backend state and side effects
2.79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error
3.wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%
4.Only three hold on to most of their pass@1 scores: GPT-6 Astra retains 78% of its single-attempt rate, and Claude Opus 5.5 and Claude Opus 5 each retain 71%
5.Kimi-K3 has the broadest coverage of any model we tested. It solves 93.89% of the benchmark at least once

Dev Signal

Get briefs like this in your inbox — free, every weekday.

100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.

Read the full issue →All briefs