0 invented answers across 120 trap questions: how we measure qbrin's trust layer
The failure that kills trust in enterprise AI is not the question it cannot answer. It is the question it answers anyway: the confident, fluent, cited-looking response about a project that does not exist, a date that was never true, a number that appears nowhere in your documents.
We call that an invented answer, and qbrin's entire architecture is organised around refusing to produce one. This post explains how we measure that claim, because a trust number nobody can interrogate is just marketing.
The setup: questions designed to bait a wrong answer
We built adversarial "trap" question sets against four separate corpora: an internal engineering workspace, the HotpotQA Wikipedia corpus, the public 37signals employee handbook, and a synthetic company workspace. Two kinds of traps:
| Trap type | What it does | Validity check |
|---|---|---|
| Invented entity | Asks about a plausible-sounding project, person, or policy that does not exist in the corpus | Code-verified: the invented name appears in zero chunks |
| False premise | Asks about a real entity but presupposes a wrong date, amount, or value | Code-verified: the false value is absent, the entity is present |
That gave us 120 distinct trap questions. Every question was generated with mechanical validity checks, not vibes: if the "fake" entity actually appeared anywhere in the corpus text, the trap was rejected before it ever reached the benchmark.
The result
Across all 120 traps on all four corpora, qbrin invented zero answers. Its behaviour split two ways, and both are what you want from a system your team relies on:
It declines. When nothing in your sources supports an answer, qbrin says exactly that. A deterministic premise guard catches many traps before a single model token is generated: if a question asserts an entity that appears nowhere in the retrieved context, the system abstains, instantly and for free.
Or it corrects the false premise, with the citation. Asked a question presupposing a film was released in 1975, qbrin answered with the source-backed fact: it was 1974, and here is the citation. On corpora where the truth is in the documents, the system pushes back rather than playing along.
A deliverable that describes an action is not evidence the action happened. The same rule applies to answers: a fluent sentence is not evidence. The citation is.
The part most benchmarks skip: auditing the auditor
A trap benchmark is only as honest as its scoring. Any answer that was not an abstention went through a groundedness audit: we checked every factual payload in the answer, every date, number, and name, against the raw corpus text. Answers that negated the false premise ("...not 1975") were classified by what they asserted, not what they mentioned. One generated trap turned out to be invalid because the "invented" policy genuinely existed in the corpus; it was excluded with the reason documented, and we kept the audit trail.
We publish the aggregate number on qbrin.com, and the raw run data, per-question rows, and the audit scripts live in our benchmark repository. If you are evaluating qbrin, ask us for them.
Why this is an architecture property, not a prompt trick
Zero invented answers is not something we ask the model nicely to do. It is enforced in layers: retrieval that only feeds real excerpts, a deterministic premise guard ahead of generation, a citation-first answer format where every factual sentence must carry its source, and verification gates that block answers whose citations do not actually support them. When any layer cannot ground the answer, the system abstains. The abstention is the feature.