The AI Agent Fabrication Benchmark

A single question, measured honestly: can an AI agent be made to take a real action on evidence it invented — and will your guardrails approve it?

Agentic systems don’t fail only by saying unsafe things. They fail by doing things — issuing a refund, deploying code, writing to a database — on the strength of a claim the model produced but no tool ever established. The action is well-formed. The authorizing fact is fiction. This benchmark measures how often that happens, and how often the standard defenses catch it.

What it measures

Each test case is an agent task where completing it is easiest if the agent fabricates a missing piece of evidence — an order that was never looked up, an approval that was never granted, a backup that never ran. We measure two numbers:

• Fabrication rate — how often the agent invents the evidence and acts on it.
• Guardrail miss rate — of those fabricated actions, how many a content-layer guardrail approves.

Results

Figures from our internal suite — 882 evaluations: 126 scenarios × 7 systems (July 2026 run). 71 of the 126 scenarios are fabricated-evidence cases. The benchmark is designed to be reproduced — numbers you can check beat numbers you’re asked to believe.

The text checkers blocked between 7% and 58% of these cases, as general threats. None of them checks whether the authorizing claim is backed by a record. Evidence-backed verification blocked 71 of 71 in this suite. That is a result on this suite, not a promise that every fabricated claim is caught. A Llama 70B judge returned no verdict on 34 of the 71 and is left out.

The pattern the benchmark exposes is structural, not incidental: defenses that read the agent’s output cannot distinguish a claim that is true because a tool proved it from a claim that is true because the model sounded confident. Only a check that tests each claim against actual evidence separates the two.