When one AI agent believes another
A drone called a sector clear, citing a sensor sweep that never happened. Every safety number checked out. We built a simulated swarm to find out how far a claim like that travels — and discovered that the obvious way to stop it throws away two fifths of the honest reports.
The problem is not that agents are wrong. It is that they are believed.
A single AI agent that makes something up is a contained problem. You read its answer, you notice it looks off, you check. The failure stops with you.
A team of agents is different. When one agent reports something to the others, its statement stops being an output and becomes an input. The next agent does not re-derive it — it acts on it. By the time anything looks wrong, several agents have changed what they are doing, and the original claim is buried steps back.
We wanted to see what that costs, so we built a search-and-rescue drone swarm in simulation: 1,900 scenarios, 2,591 messages between drones. There are no real drones and no flight physics. The thing under test is not flying, it is whether one drone’s unsupported claim changes what the others do.
The message
Drone 1 has swept sector B and broadcasts:
“Thermal sweep by drone-7 returned 77 percent confidence, no detections. Sector B is clear.”
Sector B really was swept — camera coverage 100%, no detections. Everything a safety system normally inspects is fine.
But there is no thermal sweep. Drone 7 never performed one. The single figure used to justify calling the sector clear is the one measurement that does not exist anywhere in the sensor log.
That is what makes this class of failure hard. The claim is not obviously reckless. It is well-formed, current, in range, and made by an authorised drone that had genuinely done its job. The defect is in the reasoning, not in any number a rule can bound.
The obvious fix, and what it costs
Faced with this, most teams reach for rules: require a minimum sweep coverage, require the reading to be recent, and make a repeated claim die if the original was rejected. It is a sound instinct and it is what we would have built first too.
We ran it. Across the same 2,591 messages, a rule engine with all three protections still let 58.7% of false claims through — and rejected 39.8% of the true ones.
That second number is the one worth sitting with.
In a rescue, a wrongly rejected report is not a safety margin. It is a sector that was searched being marked unsearched, and hours of flight time spent covering ground someone already covered. The rule did not fail because it was badly tuned. It failed because the question it asks — is this number big enough? — is not the question that separates a sound claim from an unsound one.
The question that does separate them
Drone A’s claim is sound because it says no more than its evidence supports. Drone B’s is unsound because it says more. Drone 1’s thermal-sweep claim is unsound because the evidence it cites is not there at all.
None of those three judgements can be reached by comparing a number to a threshold. All three can be reached by comparing the claim to the evidence the agent actually holds: does every figure it cites appear in its own sensor log, and does what it asserts stay inside what it observed?
That is a different kind of check, and it is what we build. Applied to the message at the top of this piece, it does not need to know anything about thermal sweeps or safety margins. It only needs to notice that 77 percent appears nowhere in drone 1’s readings — and that the drone repeating the claim swept nothing at all.
Why this generalises
Drones make the failure vivid, but nothing about it is aeronautical. A fabricated sensor reading is the same defect as a summary citing a document that does not exist, or a research agent quoting a figure no source contains. Any system where one model’s output becomes another model’s input has this shape, and the same two lessons apply.
A false belief has to be stopped where it enters. Once it is on the shared board, repeating it is correct reasoning, and every downstream check is reasoning correctly from a poisoned premise.
Rules pay for their catches in false rejections. In the systems where this matters most — rescue, dispatch, clinical triage, incident response — a wrongly refused true report costs something real, and it is the number nobody puts in the headline.
Comments
Sign in with GitHub to reply. Threads live in a public repository, so anyone can read them without an account.