FraudBench tests banking agent safety under adversarial dialogue

Summary

If you're building agents that execute real actions (transfer funds, reset credentials) based on conversational input, this benchmark surfaces history-dependent authorization bypasses that static transaction classifiers miss. Current safety evals don't model the cumulative effect of probes and admissions across a dialogue.

Why it matters

If you're building agents that execute real actions (transfer funds, reset credentials) based on conversational input, this benchmark surfaces history-dependent authorization bypasses that static transaction classifiers miss. Current safety evals don't model the cumulative effect of probes and admissions across a dialogue.

Sources

1. 1.the same agent that answers a question can also change contact details, reset a PIN, or move money, so ordinary customer service is inseparable from authorization, fraud detection, and policy compliance
2. 2.FraudBench contains 150 authored adversarial scenarios; a frozen public set of 107 (90 across ten fraud mechanisms plus 17 chained adaptive attacks) is used for all reported runs
3. 3.attack-security between 49% and 65%, with money-mule and first-party fraud the most common cross-model weaknesses
4. 4.Safety is history-dependent: single-control tasks satisfy every precondition but one, and adaptive attacks make a later, locally valid request unsafe because of an earlier probe, admission, or failed attempt