A 400-token cap on our verifier was silently dropping correct answers

August 16, 2026 · by Kate Sai Kishore

When your system is built to fail closed, any unhandled exception becomes a refusal. We set a 400-token output cap on our structured JSON verifier to keep latency low. On complex multi-document questions, the verifier generated detailed reasoning, truncated at token 400, crashed the JSON parser, and dropped 18.4% of fully correct answers. On the dashboard it looked like commendable caution. In reality, it was a broken parser.

One of the core design principles of qbrin’s trust layer is fail-closed execution: if a verification check cannot complete, if a network timeout occurs, or if a response cannot be parsed, the system must never assume the answer is safe. It must abstain or hold the action for a human operator.

Fail-closed is the correct architectural choice for high-stakes enterprise systems. But it creates a treacherous blind spot: software bugs and engineering bottlenecks masquerade as high model reliability.

The anatomy of the silent drop

Our claim verifier evaluates whether generated answers are entailed by retrieved passages. It returns a structured JSON payload containing a verdict, the specific source chunk IDs, and step-by-step reasoning:

{
  "verdict": "ALLOW",
  "citations": [
    { "doc_id": "sec-runbook-402", "chunk_idx": 3, "quote": "Primary feeder breaker must remain energized..." }
  ],
  "reasoning": "The operator command to maintain substation bus B voltage aligns with section 4.2 of the emergency operating procedure..."
}

To keep latency under 350ms, we configured the verifier model with max_tokens: 400. On simple factual queries, the verifier used ~120 tokens and completed cleanly.

However, when a user asked a complex question spanning four technical specifications, the verifier model produced thorough, multi-step analysis. At token 400, generation stopped abruptly mid-string:

{
  "verdict": "ALLOW",
  "citations": [
    { "doc_id": "eng-spec-881", "chunk_idx": 12, "quote": "Maximum allowable pressure differential is 45.2 psi across the secondary seal." },
    { "doc_id": "eng-spec-884", "chunk_idx": 4, "quote": "Pressure differential above 40 psi requires immediate bypass engagement." }
  ],
  "reasoning": "The proposed bypass valve adjustment is supported by chunk 4 because the current differential reading of 42.1 psi exceeds the 40 psi threshold specified in eng-spec-884, and does not exceed the

The downstream handler executed JSON.parse(response). The engine threw:

SyntaxError: Unexpected end of JSON input

The surrounding try/catch block caught the error and dutifully logged: [Verifier] Unhandled evaluation error → triggering FAIL_CLOSED abstention. The system output an honest “I don”t have sufficient verified evidence” message to the user.

The illusion of conservative caution

Because the system abstained, the error never appeared in accuracy audits. There were 0 hallucinations, 0 invented entities, and 0 security policy breaches. Across our evaluation suite, the benchmark showed an 18.4% abstention rate on complex queries, which reviewers interpreted as healthy, conservative risk management.

In truth, every single one of those 18.4% dropped queries was fully supported by the underlying documents and had been judged valid by the verifier model. The answer was discarded solely because the JSON string ran out of runway.

A system that fails closed without error distinction will make your engineering defects look like safety wins.

How we solved it

To eliminate silent parse drops without blowing out token budgets or latency, we implemented three structural changes:

1. Schema Key Ordering: In structured output schemas, place the essential scalar fields (e.g. verdict, confidence, status) at the very top of the JSON payload before lengthy prose strings or unbounded citation arrays.
2. Token-Constrained Grammar / Streaming Parsers: Use constrained decoding (such as JSON schema grammars or partial JSON parsers) that can salvage a valid verdict even if trailing reasoning strings are truncated.
3. Categorical Telemetry on Abstentions: Separate semantic abstentions (where the model explicitly returned "verdict": "REFUSE") from execution faults (parser crashes, HTTP timeouts, rate limits). If your telemetry dashboards lump all non-answers into one bucket, parser crashes will stay hidden forever.

After adjusting the token budgeting and adding streaming repair, our true abstention rate on that benchmark settled at a genuine 4.2% (true ungrounded queries), restoring valid cited answers to 14.2% of our most valuable, complex user queries.

Read more on how we measure abstentions in AI Abstention: Why Knowing When to Stop Matters and our method post on Measuring hallucination rates. Try our verified answer pipeline: Run qbrin on your complex documents →