GPT-6.1 Sol + Ling 3.1 Flash on AI Gateway

Data Point

Deterministic benchmark detects Claude model degradation

livenerf pins CLI version, freezes prompts, and runs 90-sample panels daily to statistically measure whether Claude Opus 5.5 drifts post-launch—detecting ~7.5 point accuracy changes per 10-day window.

Developers have had no objective baseline to verify whether model behavior changes after deployment. This append-only benchmark replaces anecdotal reports with timestamped statistical evidence, useful for anyone relying on consistent model performance in production.

Not a drop-in replacement for your evals—it's infrastructure for long-term monitoring of a specific model. Requires Max subscription, Claude Code CLI, Python 3.11+, and 30 days to collect baseline. Worth running now if you depend on Opus 5.5 and need to distinguish actual regressions from noise, but the first actionable results land after day 20.

• “does a model get worse after it ships”
• “runs once a day for 30 days: days 1–10 are the baseline, then there are two 10-day windows”
• “detects an accuracy change of about 7.5 points per 10-day window”
• “effort low: −62% output tokens, −8.3 ± 4.5 points of accuracy”
• “78 questions are sometimes right, and they form the panel”
• “frozen prompts, pinned CLI, exact graders, raw logs forever”