DeepsecBench ranks models finding code vulnerabilities

Summary

Developers can now compare vulnerability detection cost-to-performance across models to optimize their automated security scanning pipelines, replacing expensive frontier-model-only approaches with cheaper multi-model strategies that fit their codebase complexity and budget constraints.

Why it matters

Developers can now compare vulnerability detection cost-to-performance across models to optimize their automated security scanning pipelines, replacing expensive frontier-model-only approaches with cheaper multi-model strategies that fit their codebase complexity and budget constraints.

Sources

1. 1.the benchmark is run three times and the data published in the report is the median of the three runs
2. 2.we don't disclose the repository, the commit, the files, or the findings, so there is nothing for models to train against
3. 3.higher price does not buy proportionally more
4. 4.Kimi K3, from Moonshot AI, ranks eighth at a score of 17.56 for $12.38 on the high setting, half the top score for about a fifth of the cost
5. 5.Anthropic's most capable model, Fable 5, is absent because it declines security work, including defensive tasks
6. 6.weighting recall (R) twice as much as precision (P), because missed vulnerabilities will go unfixed, while false positives don't make your codebase less secure