AI SYSTEMS · LLM EVALS · DETERMINISTIC RULE ENGINES
Andrew Xia
The AI drafts. The rules decide.
I build AI systems with a deterministic core. My flagship reads air-cargo paperwork with an LLM, then lets a rule engine — never the model — issue every compliance verdict.
96.3% STRICT FIELD ACCURACY511 FIELDS HAND-CHECKED ACROSS 85 REAL DOCUMENTS57 DETERMINISTIC RULES · SHA-256 SNAPSHOT PER VERDICTSAME RULE IN THREE SYSTEMS ↓
Verdict — shipment pre-check, replayed
An air-cargo compliance engine I am building solo, measured against a Sydney freight forwarder's real document mix. Pick a case and watch the real pipeline's behaviour: an LLM extracts the fields, then 57 deterministic rules issue the verdict. Every flag carries a receipt.
SYNTHETIC DATA · real pipeline, invented documents
- Upload
- Classify
- Extract
- Validate
- Score
- Draft
Measured on deepseek-v4-flash at temperature 0 · prompt precheck-extract-v3.0 · 1 run (greedy decode, so one pass is the measurement)
View the eval report
Extraction eval — the shipped extractor vs hand-labelled ground truth
measured 2026-06 · corpus v1
| Corpus | 85 documents · 511 labelled fields · 17 document sets |
| Strict exact-match accuracy | 96.3% |
| Semantic accuracy (format/granularity tolerant) | 99.0% |
| Extraction failures | 0 · True misses: 2 — both reproduced by the harness and reviewed field-by-field |
| Method | Field-by-field diff, strict + lenient matchers; identifiers compared separator-stripped; every mismatch printed for human review |
| Run conditions | deepseek-v4-flash · temperature 0 · prompt precheck-extract-v3.0 · 1 run |
Numbers cover clean digital English PDFs from one forwarder’s document mix. Scanned/photographed documents and other layouts are outside this corpus — measured honesty beats round numbers.
How this demo works
This demo replays pre-recorded outputs of the real pipeline run against synthetic shipping documents — same extraction schemas, same rule engine, same verdict logic. Nothing here calls a network: no LLM API, no tracking, no data leaving your browser. Document values are invented; rule IDs and thresholds are simplified for display. The full system is still being built for that client; this replay is the honest public slice of what works today.
Demo recording: 2026-06-12 · rule snapshot b439ef74…
Systems
Only one of these has an accuracy number, because only one has a hand-labelled corpus to measure against.
Case study — Verdict
Air-cargo compliance pre-check, built on my own initiative for the freight-forwarding domain. I put an early prototype in front of a Sydney forwarder’s IT lead; he opened up their real document mix and workflows, and they are now looking at folding it into their own stack. Solo build, still in development.
The constraint that shaped everything
A compliance verdict you can’t reproduce is a liability. So the system is built on one rule: the model never decides. The LLM extracts fields, classifies documents and drafts follow-ups. Every pass / review / blocked verdict comes from a deterministic rule engine. Pure functions over validated fields, versioned with SHA-256 snapshots of the ruleset. Any verdict can be replayed bit-for-bit later.
Architecture
What decides what
| Concern | Deterministic rules | LLM | Human |
|---|---|---|---|
| Compliance verdicts (pass/warn/block) | ● | — | review queue |
| Field extraction from documents | schema validation | ● | corrections UI |
| Document classification | keyword heuristic first | fallback on low confidence | — |
| Follow-up emails | template engine | drafting assist | always reviewed |
| Risk scoring | ● | — | — |
Honest limitations
Synthetic-data demo above; the real system is in development for one client and is not a public product. Accuracy is measured on clean digital English PDFs from that client’s document mix. Not scans, not photos. Rules encode my reading of public regulations, verified by tests, not by a licensed broker. No production-traffic claims.
Roadmap (building · designed)
- building HS-code classification: deterministic layer running (keyword matcher over a 5,613-code AHECC dataset, plus tariff and FTA lookup, behind three endpoints). The AI-assisted path above it is a stub that returns nothing, and there is no UI
- designed compliance copilot for explain-why queries (read-only)
- designed email-thread intake orchestration
Status verified 2026-06-12
The problem it started from, the full spec and the provider trade-off →
How I build with AI
Verdict was built with an AI pair I governed like an untrusted worker. What began as five documents and a set of hooks is now DevLoop, a tool in its own right.
- Spec first.
Five source-of-truth documents and a 17-stage prompt pipeline define every change before the model sees it.
- Guard hooks.
Custom pre-execution hooks block dangerous commands and out-of-contract edits before they happen.
- Read-only reviewers.
Separate subagents audit diffs against spec. They can flag, but they cannot write.
- Gates, not vibes.
Pre-commit runs type-check, tests and build; CI re-runs everything. Green is the only accepted evidence.
- Failure-mode dictionary.
10 documented incident patterns, each mapped to the safeguard that now catches it. One of them is the model reporting an exit code for a command that was not in the hook it named.
Writing
Notes from building under the assumption that the model is lying.
Contact
Questions about how any of this was built? The fastest way to see it working is the demo above; the second fastest is twenty minutes and a screen share.
xhh-0-hhx@outlook.com LinkedIn
Email reaches me directly.
Ask for a walkthrough