We built a benchmark that said we were perfect. Then we tried to break it.

August 10, 2026 · by Kate Sai Kishore

Our first run showed a rival agent hallucinating on 9.2% of questions and our own system hallucinating on none. An independent audit of our own harness found fifteen defects, twelve of them tilted in our favour. After we fixed them the measured hallucination rate was zero. Every one of those “hallucinations” had been ours.

We wanted to know something specific and unglamorous: when a language-model agent is pointed at a real security index and asked ordinary analyst questions, how often does it state a number that is not true?

So we built the thing properly. A real Splunk Enterprise install — version 10.4.2, build 33c3bf42cd73 — on a real server, indexing 384,782 real events off that host: three weeks of Wazuh alerts, Linux SSH and PAM logs, nginx access logs, PostgreSQL, syslog. No synthetic specimens and no replayed sample data. Then 227 questions generated from templates over values actually present in the corpus, with every expected answer computed from Splunk itself rather than typed by hand.

The first result looked excellent for us. The agent fabricated on 9.2% of the questions it answered. Our deterministic system fabricated on none.

That number is why this post exists. It was wrong, it was wrong in our favour, and we would have published it.

The tell was that it was too clean

A 9.2% fabrication rate against a 0% rate is the kind of result that ends an investigation instead of starting one. We had already shipped one scoring bug earlier that week which quietly flattered us, so before writing anything up we did the obvious thing: we pointed three independent reviewers at our own benchmark and told them to break it. One was briefed specifically to argue the opposing system’s case, and to find every way our harness handicapped it.

They returned eighteen claims. Fifteen survived verification against the actual code.

Twelve to one is not noise. It is the signature of a measuring instrument built by people with a preferred answer.

Four ways a benchmark lies for you

The defects were not exotic. They are the kind any team writes, and each is invisible from the inside because it only ever produces results you were already expecting.

1. The scorer had learned our own vocabulary

To score a question about something that does not exist, we checked whether the answer said so. The first pattern in our detector was does not appear.

That is the literal opening phrase of our own system’s hard-coded refusal template. Our system matched it every time, by construction. The agent, writing free prose, had to land inside the same closed vocabulary and mostly did not. “Does not exist”, “no matching events”, “cannot be determined”, “was not observed” all missed. Eight of eleven natural ways to correctly report absence were scored as fabrication.

2. The number checker read the question back as an answer

To catch invented counts we scanned answers for numbers. When a correct answer echoed the IP address from the question, 203.0.113.45, our scanner read the leading octet and recorded it as a claimed count. The scorecard would have reported “claimed 203, truth 0” about a fabrication that never happened.

We fixed that by masking identifiers echoed from the question. It bit us again anyway, in a form we had not anticipated. Asked what condition a nonexistent Wazuh rule detects, the agent answered:

Correct answers cite real numbers as supporting context. A checker that treats every number as an assertion cannot tell the difference between a measurement and an illustration.

3. Ground truth was our own query

This was the structural one. We computed each expected answer by running a Splunk query, and our own system answered by running a query too. The audit normalised both sets and diffed them: 29 of our 34 operations were character-identical to the ground-truth query.

Our system could not disagree with the truth, because it was the truth. Its perfect score on that stratum measured nothing at all. The same circularity held for the trap questions: the gate that admitted a trap into the question set was the same probe our system used to detect one.

4. The harness manufactured the failure it was measuring

Three defects in the test rig itself, each producing agent errors that had nothing to do with the agent:

• Query results were truncated at 100 rows with no marker. The agent reasoned over a silently shortened result set, reported a confidently wrong total, and we scored it as a hallucination.
• The turn limit cut off two of four trap questions while the agent was already holding the correct evidence. It had received an empty result five separate times and was cut off before it could say so. It was punished for being thorough.
• Our tool description spelled out which fields were extracted for the one data source where they are, and said nothing about the four where they are not. The agent reasonably generalised, filtered on a field that does not exist, received a valid count=0, and reported zero.

Fixing the benchmark deleted the finding

We corrected the scorer, rebuilt ground truth with an independently written second query per template, announced truncation, raised the turn limit, and added a forced final answer so that a harness timeout could never be recorded as a fabrication. Then we re-scored the answers we already had. The agent’s output never changed; only the instrument did.

On the full re-run with a corrected harness the agent answered 226 of 227 questions correctly and fabricated nothing at all on the false-premise questions. It correctly reported every nonexistent rule, agent and CVE, and never invented an attacker.

The finding we did not expect: most of these questions have more than one right answer

Rebuilding ground truth with a second, independently written query per template exposed something we think is badly underappreciated. 57 of 127 cross-verified questions had two defensible answers.

Ask how many login attempts used the invalid username admin — a question with an obviously exact answer. SSH writes one such attempt as three different log lines:

Our ground truth said 21. The agent said 56. We had recorded that as a hallucination. Both are real counts of real events and the question does not distinguish them.

It gets worse in the direction that should worry anyone grading a model. Asked how many requests hit the path /, the agent answered 1,098, counting every HTTP verb. Our ground truth said 1,087, because it counted only GET, POST and HEAD. The agent was more correct than the answer key, and our scorer called it a fabrication.

What we would tell you to do

None of this needed special tooling. It needed assuming the result was wrong and then working to prove it.

1. Have someone adversarial read your scorer, briefed specifically to argue for whatever you are measuring against. Every defect we found was invisible from inside the team that wrote it.
2. Compute ground truth by a route the system under test does not share. If your answer key and your product run the same query, you are measuring a tautology. Where two valid formulations disagree you have found an ambiguous question, not a hallucination.
3. Read the raw answers. Both of our worst bugs were invisible in the aggregate and obvious the moment we looked at the text a verdict was attached to.
4. Never let your harness’s own limits score as a model outcome. Truncation, turn caps and provider errors are your failures, not the model’s. Provider errors now get their own outcome in our scorer, excluded from every denominator; a run that hit an exhausted API account is reported as void rather than as a model that stopped answering.
5. Be most suspicious when a bug flatters you. Twelve of fifteen went one way. That ratio is the signal.

One last detail, because it is the reason the corpus is frozen. An earlier run scored against a live log file and one count moved underneath it mid-run, 1,823 to 1,830, as real logins arrived. Scored naively, that would have been an error the model never made.

We set out to measure how often an agent invents a number. What we actually measured, for most of a week, was our own instrument. The useful part is not the final figure. It is that the first figure was confident, quotable, entirely wrong, and pointed in exactly the direction we wanted it to point.