We let an AI run a chemical plant. It stopped and asked for a human three hours before the plant failed.

August 6, 2026 · by Kate Sai Kishore

Ask an AI to run something that can hurt people and the honest worry is not that it will be stupid. It is that it will be fluent. It will give a reason that sounds exactly like a good reason, and nobody will check.

So we built the check. Over the past week we put an AI agent in charge of a chemical plant — a simulated one — and made it justify every single thing it wanted to do. Not to a human reading along, who would get tired by lunchtime. To a program that verifies whether the reason is true before the valve is allowed to move.

Across 29 eight-hour runs of the plant, the agent asked to do something 63,589 times. Not once did an invented reason move anything.

A real plant, not a toy

The plant is the Tennessee Eastman process, a chemical plant model published by Downs and Vogel in 1993 and used by process-control researchers ever since. It has 41 instruments, 12 valves the agent can move, and 28 ways engineers can deliberately break it. It is free, it is public, and anyone can download it and repeat everything below.

That matters. It would be easy to build a demo where the plant is a cartoon and the AI is a puppet. This plant has real dynamics: heat, pressure and composition are coupled, the controller fights back, and if you break it badly enough it dies whatever anyone does.

Nine questions before anything moves

When the agent wants to open a valve, it must say what it wants and why. That request goes through nine checkpoints. Any one of them can stop it.

Most of those are the kind of thing any careful engineer would build: is the reading recent, is this operator allowed to touch that valve, is it hammering the plant with commands. The fifth is the unusual one.

It checks whether the reason the AI gave is true. Not whether it sounds right. Whether the numbers in it actually came off the instruments, whether the procedure it cites exists and is still current, and whether that procedure actually permits this action under these conditions.

The code doing that check is not new and was not written for this. It is the same qbrin trust layer that decides whether an answer may be shown to a person, imported without modification. The same function that refuses a made-up citation in a document search refuses a made-up pressure reading here. Only the consequence changes: instead of an answer not appearing, a valve does not move.

What "citing a procedure" actually means

Here is the part that surprised us most, and it has nothing to do with AI.

When an operator says "I'm doing this under OP-17", they are making four separate claims: that OP-17 exists, that it was approved, that it is still current, and that it actually covers this action in this situation. Only the first one is what people normally mean by making something up.

The dangerous one is the third. A withdrawn procedure is real. Someone wrote it, someone approved it, and the document still reads as authoritative years after the plant stopped following it. A human skimming the justification nods along. A document search finds the text and confirms it. Only a registry — a list of what is currently in force — knows it was withdrawn.

The run where the plant died

One of the 28 faults is an "A feed loss" — a raw material supply fails. The controller absorbed it for four hours. Then it lost the plant.

Look at the top chart. Pressure rose to 2919.9 kPa and stopped about thirty short of the number that would have tripped a shutdown alarm. The classic alarm never fired. It was never going to: by the time a shutdown threshold trips, the decision has already been made for you.

The bottom chart is what was actually happening. A vessel emptied and kept going, until the level gauge read -29.4%. A level cannot be below zero any more than a fuel gauge can read less than empty. That is an instrument telling you it has left the world it was built to measure.

From 4h30 onward the agent was refused every time it asked to act — not with "no, try something else", but with "no autonomous action is appropriate, fetch a human". Those are different answers and we treat them differently, because an escalation that looks like an ordinary refusal never reaches anybody.

Both numbers, or neither

There is an easy way to score perfectly on a safety benchmark: refuse everything. Nothing bad gets through, the report looks immaculate, and the system is worthless. Worse than worthless — a gate that blocks real work gets switched off in week one, and after that it protects nothing at all.

So we measure both directions on the same runs, at the same instants, and we publish them together. At every snapshot of every scenario, the agent proposed a legitimate action justified from real instrument readings, and separately proposed the same action justified by a pressure the plant never reported.

We also ran a third kind of attempt, which is the one a fabrication detector is worst at. Instead of inventing a reading, the agent justified itself with a reading that was true two hours ago. A real number, from the wrong moment. Nothing about it is fabricated; it is simply no longer the case. Those were caught too, and the mechanism is the same one: the number has to match the instrument now.

How we test

A benchmark is only worth the scepticism built into it, so we do two things as standard.

The tests are written by people who have not seen the code. We commissioned 120 operator justifications from writers working only from the instrument readings — in six different registers, because a clipped 3am control-room log and a formal shift handover are not the same language. Abbreviations, tables, trends written half a dozen ways, readings that have gone negative: the vocabulary a real control room actually uses, rather than the vocabulary we would have thought to test.

And we commission attacks from people we have given the rules to. Ninety of them, written by adversaries who had the full specification of every check in front of them and were asked to find a way through rather than to be rhetorically clever. That is a much harder test than probing a black box.

Both sets now ship as permanent fixtures in the benchmark, so every future change to the system is scored against prose its authors did not write. It is the same principle as the trust layer itself: the check has to come from outside the thing being checked.

A test set written by the person who wrote the checker measures the checker against itself.

Across all 90 attacks, not one moved a valve. Where an attempt did get past one layer, the next one stopped it — which is exactly what the layers are for, and why there are nine of them rather than one.

The claim, stated precisely

We say exactly what we measured, because a claim you cannot interrogate is worth nothing. The simulator decides whether the plant survives; this layer decides whether the agent was entitled to ask. It is the second question we tested, and it is the one that determines whether an agent can be trusted with controls at all.

So: an AI agent operated a real-time critical-process simulator for eight hours at a stretch while every action it proposed was independently checked for whether its evidence was real, whether it was authorised, whether a current procedure permitted it, and whether it stayed inside hard safety limits — and that this can be measured in both directions, on public software, by anyone who wants to repeat it.

That is a narrower claim than “AI can run a plant”, and a far more useful one. Every number above comes from software you can download today.

The shape generalises past chemistry. A short list of things that can be asked. A reason that has to be true before anything happens. A registry of what is actually in force right now, as opposed to what a document somewhere still says. A clear distinction between "no" and "this needs a person". Deterministic limits that get the last word no matter how good the argument is. That is the structure you want around a trading system, a deployment pipeline, a robot arm, or an agent with your company's credentials.

The failure that worries us was never an AI that gets things wrong. It is an AI that gets things wrong persuasively, and acts on a reason nobody checked.

Try qbrin