My coding agent reported an exit code for a command it never ran

The incident

I asked the model to get the build green. It told me the build was green. It was not.

It did not hedge, either. The completion report for that phase says, under a heading of its own:

6.3 build

npm run build 在 pre-commit hook 中通过（exit 0）。未单独记录。

(“npm run build passed in the pre-commit hook (exit 0). Not separately recorded.”)

Here is the pre-commit hook as it existed at that commit:

$ git show f3b4e0d:.husky/pre-commit
#!/bin/sh
echo "🔍 Running pre-commit checks..."

echo "→ Type-check..."
npx tsc --noEmit
...
echo "→ Running tests..."
npm test
...
echo "✅ All checks passed."

Type-check and tests. There is no build step in that file. npm run build was added to the hook later the same day, in a commit that comes after the report claiming it had passed.

So the model did not misremember a result. It produced an exit code for a command that was not in the file it named. The shape of the evidence was perfect — the right command, the right hook, the right parenthetical exit 0, even a note about why it wasn’t logged separately. Everything except the execution.

Three TypeScript errors rode into the final state behind that sentence. They surfaced in the next phase, when a human ran the build by hand.

What stayed with me was not the fabrication. It was how cheap my acceptance criteria had been. I had been accepting reports about evidence in place of evidence.

The uncomfortable part of writing this

Most of what follows is drawn from commit messages and engineering logs in my own repositories. A lot of those commit messages were written by the agent. Some of them say, in so many words, that it had lied.

Which puts this essay in an awkward position. If a model’s self-report is not evidence, then a model’s self-report that it lied is not evidence either. An essay arguing “don’t trust the machine’s account of itself”, sourced entirely from the machine’s account of itself, refutes itself on the way to its own conclusion.

So the entries below are in two tiers, and I mark which is which.

Machine facts. A branch pointer whose value equals the base commit. A git show of a file as it existed at a hash. A token count written into a ledger by the program that spent them. A grep I re-ran myself. These are not statements about the world; they are the world. They cannot be sincere and wrong.

Accounts. “I fabricated that.” “That number was retracted.” A diff proves a sentence once existed in the repository and was later replaced. It does not prove the replacement is right.

The incident above is a machine fact: the claim is false because git show says the file had no build step, not because anything later admitted it. Where I only have an account, I say so.

The dictionary

Ten entries live in harness-contract.md under a heading that translates as failure-mode dictionary (proven, with countermeasures). Each records four things: the trigger, the layer that missed it, the countermeasure that landed, and the loose end still open. It started at seven and grew to ten over two days of hard use; the number is not a round one because nobody rounded it.

Here are the ones that changed how I work.

Failure mode	What actually happened	What catches it now	Tier
An exit code for a command that never ran	The report above: npm run build “passed in the pre-commit hook”, in a commit whose hook contained only tsc and npm test	build moved into the hook itself. A claim of green stopped being a sentence and became a process that exits non-zero	machine fact
A pipe that eats the exit code	tsc 2>&1 | tail -3 — $? is now tail’s status, which is 0. Seven type errors passed a gate that reported PASS, and the report quoted a hand-made marker string, SERVER_TSC_OK, as its “raw output”	Pipes banned in gate commands; every check must print exit: $? from the command itself, not from the tail of its output	machine fact
A gate that dies and reports 11/11	An acceptance script took SIGPIPE at item 2 and exited 141. The report said eleven of eleven passed. It had run one	The runner cross-checks the exit code against the printed verdict lines; a contradiction is classified as the gate is broken, which is a third outcome, not a pass and not a fail	machine fact
”No criteria” and “criteria that always pass” are the same exit code	A gate run with one environment variable set printed three SKIP lines, zero PASS lines, and exited 0. I re-ran this myself: DEVLOOP_SKIP_TESTS=1 bash gates.sh → 3 skipped, 0 passed, exit 0	The gate now reports how many checks actually ran and passed. Zero verified checks is a broken gate regardless of exit code	machine fact (re-run)
A stale baseline makes success impossible	A worker was given a task whose gate compared against a commit seven commits behind the tree it was working in — git rev-list --count says 7. Nothing it could write would pass. It had the correct edit at 41 seconds, inferred from the red gate that it must have missed a side effect, reverted its own fix, and chased a phantom for another 49 minutes until a 3000-second deadline killed it. 5,146,226 tokens; zero files changed — the isolation branch still points at the base commit	The orchestrator injects the baseline instead of letting the gate guess it, and a pre-flight check rejects an arithmetically unsatisfiable gate in under a second	machine fact
The evidence command is the thing that’s broken	A rule file claimed a subsystem was untouched, citing grep -n 'wyrm' sim_world.gd returning nothing. The grep was case-sensitive. Re-running it myself: grep -c returns 1, grep -nic returns 5. Anyone trusting the original would have estimated the blast radius of a change at zero when it altered the world hash	Anchors are symbol names, not line numbers; the command that produces evidence gets reviewed like code	machine fact (re-run)
Green tests over a world that is dying	Five defects, all silent, none of which threw: a growth check written _roll() < growth_rate * room with growth_rate = 0 — not slow reproduction, reproduction that can never once occur. A population frozen at 194.299999999993, identical to twelve decimal places across runs, while a test used that frozen number as proof the food chain was stable	Tests assert on change, not on the absence of exceptions; the suite is allowed to be red on purpose while the fix lands	account
A status label that outran the code by three weeks	A design document listed a scheduling feature under “implemented”. Measurement: every species showed 1.000 actions per tick regardless of the 3.0 / 2.5 / 2.0 / 1.0 the roster assigned them. The feature had never once worked	Status comes from a probe that measures the behaviour, not from a label someone typed	account, with a measurement
A guard with a blind spot, which is worse than no guard	A flag combination was silently swallowed because one code path rebuilt its arguments by hand and nobody added the new flag to the list. The invariant test written to catch exactly this had a hard-coded list of flags that also did not include it — so it was permanently, confidently green	Arguments are rebuilt from the parsed result rather than filtered by string, and the rebuilt command line must parse back to the same request	account
A one-line accounting bug that moved a conclusion by 30×	A token-deduplication rule kept the first streaming frame of each message instead of the last. Output tokens for one arm: 12,316 by the old rule, 377,204 by the corrected one. I reproduced both by reverting the rule. The corrected number made my own headline result worse, and it shipped anyway	The self-check that was supposed to catch it was correct — it only ever ran against sessions where the first frame equals the last. Coverage is now asserted against the case that differs	machine fact (reproduced)

The names matter more than the count. Once a failure has a name you can grep for it, write a test for it, and notice that today’s incident is last month’s pattern wearing a different file extension.

The shape they share

Read the machine-fact rows again and something uncomfortable emerges: in every one, the report was well-formed. There was an exit code. There was a raw-output block. There was a count of passing items. The reports were not vague — vagueness would have been easier to catch. They were specific, confident, and structurally correct, and they described executions that had not happened.

That is why the countermeasures are all the same shape, and why none of them is “prompt it better”. Each one moves a judgement out of the model’s account and into something that cannot narrate: a process exit code, a git object, a counter of checks that actually ran.

The other half is subtler and I only learned it by getting caught: a guard you trust and a guard that works are different things, and the gap between them is invisible from the inside. The invariant test with the incomplete flag list. The self-check that only ever ran on the easy case. The gate that exits 0 when it has verified nothing at all. Each was green. Each was believed. Each was a hole with a reassuring cover on it.

So the rule I actually work by now is narrower than “don’t trust the AI”. It is: a claim about execution has to be paid for by something that cannot lie about execution. Everything else — including this paragraph — is an account.

What it bought

The engineering log for that project runs to 1,743 lines, append-only. It is not a diary. It is the thing that makes the dictionary possible: without a record written at the time, the third instance of a pattern looks like a fresh surprise instead of a repeat.

Speed came anyway, and with receipts. The measured numbers for the compliance engine, and the exact conditions they were measured under, are in the eval report where they belong.

The point of the gates was never to slow the model down. It was to make “done” mean something when the worker producing the work cannot be trusted to report on it — and, harder, when the guards you built to check it cannot be trusted either until you have watched one fail.