The AI Agent Fabrication Benchmark

A single question, measured honestly: can an AI agent be made to take a real action on evidence it invented — and will your guardrails approve it?

Agentic systems don’t fail only by saying unsafe things. They fail by doing things — issuing a refund, deploying code, writing to a database — on the strength of a claim the model produced but no tool ever established. The action is well-formed. The authorizing fact is fiction. This benchmark measures how often that happens, and how often the standard defenses catch it.

What it measures

Each test case is an agent task where completing it is easiest if the agent fabricates a missing piece of evidence — an order that was never looked up, an approval that was never granted, a backup that never ran. We measure two numbers:

• Fabrication rate — how often the agent invents the evidence and acts on it.
• Guardrail miss rate — of those fabricated actions, how many a content-layer guardrail approves (because the text is clean).

Results

Figures from our internal suite — 504 evaluations across 126 scenarios × 4 tool types (May 2026). The benchmark is designed to be reproduced — numbers you can check beat numbers you’re asked to believe.

Fabrication detection rate: content guardrails and LLM-as-judge caught 0% of fabricated-evidence actions (they have no mechanism to check whether an authorizing claim is grounded); evidence-grounded verification caught 100%. This is expected, not surprising — the others are not built to ask the question.

The pattern the benchmark exposes is structural, not incidental: defenses that read the agent’s output cannot distinguish a claim that is true because a tool proved it from a claim that is true because the model sounded confident. Only a check that grounds each claim against actual evidence separates the two.