Designing AI workflows with explicit boundaries

My working method: define the decision, separate implementation from acceptance, and make uncertain outcomes visible. The examples below show which checks run automatically, which need an operator, and which remain judgement calls.

The aim is useful software that people can operate and question—not more process for its own sake. Checks reduce specific risks; they do not guarantee a useful product or correct reasoning.

Method notes: assumptions, historical failures & limits

Working assumptions

These are the assumptions I work with, stated so you can disagree with them:

  1. A new model call can use only the context made available to it.

    Conversation history, saved notes and memory tools can carry earlier work forward, but only when the workflow retrieves and supplies them. A convention stored somewhere is not necessarily present in the next decision.

  2. AI can produce changes faster than I can validate them.

    A draft can arrive quickly; understanding its effects, testing its boundaries and deciding whether it solves the right problem still take work. Faster generation does not remove that work.

  3. Errors have asymmetric costs: making one can be cheap, finding it expensive, and fixing it after other work depends on it more expensive still.

    Optimising only for output can shift work into later diagnosis and repair. The value of faster generation depends on the work needed to check and correct the result.

Design choices

These assumptions, together with the incidents below, inform the following design choices:

  1. A check is only as independent as the boundary protecting it.

    A model’s self-review is not independent acceptance evidence. I also use rule engines, process exit codes and separate implementations. Their results still depend on the checks and configuration: a recorded probe disabled a guard by adding a trailing comment to the line it read, while it continued reporting zero violations.

  2. Faster generation makes verification more important.

    Building a useful product still matters. Alongside implementation, I invest in defining what counts as done, checking the result, and testing whether the checks can actually reject a broken example.

  3. Short feedback loops can catch errors before later work builds on them.

    Cheap checks move closer to the tool call so mistakes are caught before more work depends on them. But hooks see only the operations they inspect: two shell one-liners went through the same hook and the files were really written. That channel is checked with before-and-after snapshots instead—detection, not prevention.

  4. Intent needs an execution path, not only a place to be written.

    A rule in a document does not run itself. Where possible, I pair it with an executable check. Programs and configuration can also be changed or bypassed, so the check’s own inputs and execution need review.

  5. Repeated failures without a visible signal can accumulate unnoticed.

    Silence is not a signal, so nothing throttles it. A cleanup routine captured output without checking the return code. The recorded incident accumulated 2,983 leaked worktrees (about 26 GB), plus about 60 GB of pytest temporary files: roughly 86 GB in total. Failures now have to be visible even when nobody is watching.

What it does not solve

What it does not solve — measured, not anticipated. An earlier version of this list was written before the work started and read better; these are the ones that turned up afterwards:

  • A check can test a claim against specified evidence. A pass does not validate every step of the reasoning behind it: an explanation can remain untested when the check measures something else.

    COSTOne bad estimate of a root cause sat behind a green board for twenty-six commits. The existing checks did not test that explanation; a targeted measurement broke the deadlock. Choosing what to measure required judgement. Once the measurement is defined, its execution may be automated. A green result still needs a clear account of what was tested.

  • A passing status alone does not show what a check examined. A text-matching check can lose its target when wording changes; without a separate coverage check, an empty match can be mistaken for a clean result.

    COSTIn the recorded audit, six of eight checks had this defect. Changing two words in a comment made an over-budget document quietly compliant. The current status-facts gate checks for missing assertion classes and rejects a document with no checkable assertions. That addresses this failure mode within its supported formats; it does not prove complete coverage.

  • A check whose switch is reachable by the thing it checks. Adding a trailing comment to one line of a protected-paths list made the pattern match nothing at all — so a file could be edited and the guard watching that exact file reported zero violations. One edit did the trespass and hid it.

    COSTOpen in the cited historical record; current resolution is not asserted here. The proposed fix needs every pattern to match an existing target, and intentionally empty patterns to be explicit. Half a fix could restore confidence without restoring coverage.

  • A historical nonconstant audit on 22 August 2026 counted twenty-nine protocol rules: five classified as mechanised, eleven as implementable gaps, twelve as judgement boundaries and one permission. That classification even included a human-checked requirement. It is a record of that phase, not a count of today’s installed gates or a claim that every labelled control runs automatically.

    COSTPublishing the classification exposes how much still relied on judgement or unfinished work. The later README records five of thirty-three protocol rules at its own dated measurement point; the protocol and the ten registered gate scripts are different inventories.

  • Hardening the checks is optimising for a user who does not exist yet, and there is no natural stopping point — a checking system can always be made more robust.

    COSTSeven consecutive working sessions went entirely into meta-work and moved the one thing that would produce a user precisely nowhere. There was a written threshold for this: meta-work above fifteen per cent triggers a cleanup. It ran at roughly a hundred, and nothing was checking the threshold. The rule against unchecked rules had no check.

These arguments grew from the working notebook now published with nonconstant, with examples from my other projects. Agents helped write those notes; their first-person accounts are not automatically my own observations. My contribution is the decisions, acceptance criteria and review, including rejecting earlier conclusions. Historical figures refer to the recorded phases, not current totals. Public source makes a claim inspectable; it does not by itself prove that the claim is correct.

PUBLIC CHECK & LOCAL RELEASE CHECKS

The public rules digest can be checked without this source repository, using curl and a SHA-256 tool. The other two commands document checks I run in this site’s private checkout.

  • curl -s https://hh0hh.com/rules.json | sha256sum Hashes the public rule catalogue for comparison with the demo’s snapshot identifier. This verifies which rule bytes were served, not whether the rules or findings are correct.
  • npm run release In this site’s private source checkout: blocks release on a stale rule digest, out-of-date share card or unresolved output placeholder. It is not a command a visitor can run without that source.
  • node scripts/check-placeholders.mjs --release Also requires the private checkout. This runs the placeholder check by itself; a recorded negative probe restored a violation to dist/ and confirmed exit 1.

Examples from my projects, with the automation level stated for each:

These labels show how a control runs: automatically, on request, or as a stated practice.

  • ENFORCED In the configured project or host, the check runs on a commit, CI job or tool call. Its stated conditions and coverage still apply.
  • RUNNABLE A real criterion that really fails — but someone has to type the command.
  • STATED Followed, but unchecked. If I stopped, nothing would go red.

PROJECT EVIDENCE & ACCESS

nonconstant’s verification code and design notebook, and the historical DevLoop source, are public. Verdict and Terrarium source remain private; their examples below are author accounts with the public demonstrations and recorded evidence available on this site.

Different checks answer different questions

Type checks constrain interfaces, host hooks control selected commands, and a separate implementation supplies a reference comparison. These mechanisms answer different questions; an execution claim still needs a record of what ran.

Verdict

The rule engine is synchronous by contract. Adding `async` fails the call sites at compile time, and the pre-commit hook stops the commit. ENFORCED

DevLoop

When its host hooks are registered and working, a regex sets a mode token and selected quota-spending commands require it. Missing or broken hooks can fail open. ENFORCED

Terrarium

A separate numpy implementation supplies checkpoints and structural comparisons. It is distinct from the repeated-engine fingerprint test and does not require identical long-run populations. RUNNABLE

The essay faces an awkward version of the same problem: much of its material comes from machine self-reports. It distinguishes file and run records from accounts, so readers can see what each claim rests on.

Failure cases test the checks themselves

A guard I trust and a guard that works are different things. Deliberate failure cases help test the difference, within each check’s stated scope.

Verdict

`build` sits inside the hook, because a report once said it passed there — in a commit whose hook had no build step. ENFORCED

DevLoop

A gate counts the checks that ran and passed. Zero verified checks is a broken gate — a third outcome, not a pass. ENFORCED

Terrarium

Ten test files declare expected assertion counts and compare them with the executed total, so a test that stops early cannot pass merely by printing no failure. RUNNABLE

One recorded probe failed to apply its deliberate break, so nothing was broken and the green result proved nothing about rejection. That is why I check that the intended failure actually occurs. It is still not enough: a probe that fails to start also exits non-zero. One recorded round of four red checks turned out to be four startup failures.

“I didn’t check” may never render as “it’s fine”

Not-checked and checked-clean are the same exit code until you force them apart. So a verdict that does not name what it skipped is itself a false green.

DevLoop

A constitution registering no unjudgeable clause refuses to load. Not a warning — a config error, with the reason in the message. ENFORCED

Terrarium

Unjudgeable clauses are a table, and every verdict carries an “N not covered” tail. Inventing a heuristic for them is banned. RUNNABLE

This site

The evaluation-conditions row renders only when model, temperature, prompt version and run count are present. The score itself is separate; this checks metadata presence, not truth. ENFORCED

Verdict carries the written counterpart: a list of what its automation cannot verify — regulatory semantics, prompt adherence after a model upgrade, credentials committed by hand.

Separate implementation from acceptance

Reading the gate and planning to route around it are indistinguishable byte for byte, so where it works the answer is to make the route-around ineffective rather than to detect intent. Where it does not work — anything that reaches the filesystem by a path the guard never sees — the honest fallback is detection, and saying so is cheaper than pretending the guard is a sealed box.

DevLoop

The gate runs from a fingerprinted copy outside the worktree. Caches rebuild, and acceptance uses a clean checkout to identify the tested revision—not to guarantee its correctness. ENFORCED

Verdict

Reviewer profiles declare a read-only tool set. Actual enforcement depends on the host configuration; the declaration is not a security sandbox or proof of execution isolation. STATED

Terrarium

Tests and answer keys are treated as protected criteria. Changing a baseline needs review: a rewritten answer key can produce a normal-looking score. This is not filesystem isolation. RUNNABLE

Every conclusion ships with its range

A result without its scope is not a result. What it covers, what it explicitly does not, and what evidence would overturn it — and a qualifier one click away from its claim is not a qualifier.

Terrarium

Four fields per ruling: what it covers, what it does not, the basis, and what would overturn it. The last two are the ones that do the work. STATED

Verdict

Two recorded runs of one document set differed. I distinguish rule evaluation on fixed validated inputs from model extraction; end-to-end reproducibility was not established by those runs. STATED

This site

A ledger row may state a mechanism and never a score, because a score needs its badge, its scope note and its limitations beside it, and one line has room for none of them. STATED

I publish corrections to my own claims

Corrections ship even when they make the headline worse, and an unknown cause is written down as unknown rather than given one.

DevLoop

A one-line deduplication bug moved output tokens 30×. The correction turned my own headline result from a win into a loss. It shipped. STATED

This site

This page once printed a rule ID, a hash and a PASS in its source. All three were invented. It is now a digest you can recompute. ENFORCED

Terrarium

A 20× speedup is on file as a rejection: faster, but not the same world. Wrong history is struck through, never deleted. STATED

Verdict keeps a section titled “what was explicitly not built”, listing the features whose interface exists and whose backend does not — including the one a demo could most easily gloss over.

← Back to the site