Andrew Xia · https://hh0hh.com
Case study — Verdict
Verdict is an independent prototype for reviewing shipment paperwork. I designed the workflow, system boundaries and evaluation, using AI-assisted implementation. It brings conflicting document fields and their supporting evidence into one review flow.
The problem
Shipment information arrives across air waybills, invoices, packing lists and declarations. A reviewer has to spot missing documents and conflicting quantities, weights or parties, then prepare a clear handover. I built a workflow that brings those comparisons together and keeps the supporting details visible.
Designed around the review, not just the model
Access and change boundaries
Authentication is followed by case-owner checks. Completed pre-checks must be reopened before reviewed inputs can change.
Failures stay local to a file
Bounded retries and per-file failure handling keep one extraction failure from discarding the whole batch.
Findings lead back to their basis
Findings identify the rule, documents, fields and values behind a conflict. A person verifies them and reviews follow-up drafts before acting.
My central design decision
I separate model extraction from rule-based decisions: the model proposes fields; deterministic rules assign the system’s review status and findings. Reviewers verify the sources, correct values and decide the next action.
Architecture
-
Input
Documents
Bring documents together while retaining field sources.
-
Model-assisted extraction
Propose fields
The model extracts fields. Extracted values can still be wrong.
-
Deterministic core
Check structure
Validate shape and types, not whether the content is true.
-
Deterministic core
Apply written checks
Written checks assign system status and identify findings.
-
Human judgment
Human review
A person checks evidence and gaps, then decides what happens next.
Assisted drafting · outside the deterministic decision core
Templates and model assistance prepare follow-ups for review before sending or acting.
Traceable records
- audit log
- field corrections
- document versions
Records make inputs and changes inspectable; they do not establish truth.
Explore implementation details
Schema validation checks structure, not whether a field is true. Compliance-review reports use SHA-256 to identify a snapshot of rule metadata. That snapshot records rule descriptions, not the executable conditions or proof that a result is correct.
- Document classification
- heuristic classifier → LLM fallback
- Model extraction
- JSON-mode extraction
- Structure validation
- Zod schema validation
- Deterministic checks
- (57 across compliance review and shipment pre-check)
- Rule output
- review status + findings
What decides what
| Task | Deterministic logic | Language model (LLM) | Reviewer |
|---|---|---|---|
| Assign a review status | Assign pass, review or block | — | Verify findings and decide next steps |
| Extract document fields | Check structure and types | Propose field values | Check and correct values |
| Identify document types | Match keywords first | Assist when confidence is low | — |
| Prepare follow-up emails | Apply templates | Assist with drafting | Review before use |
| Calculate a rule-based score | Calculate the score | — | Interpret alongside findings; not a probability |
Recorded verification
The prototype documents 9 document types and 57 checks across compliance review and shipment pre-check. Project records list approximately 1,000 test cases and 8 adversarial test rounds. CI checks tests, types, build and lint; these checks do not cover every real-world situation.
A separate June 2026 evaluation made one recorded pass over clean English text reconstructed from 85 synthetic documents in 17 sets. Extracted values matched the expected values in 96.3% of 511 fields after normalisation and numeric tolerance; allowing substring containment raised agreement to 99.0%. PDF parsing and OCR were not evaluated.
Data boundary & provider choice
Model calls sit behind a provider interface, separate from the decision engine. DeepSeek and MiniMax are implemented and selected by configuration, with bounded retries within a provider—not automatic cross-provider failover. A new backend requires an adapter and validation. The public examples use synthetic data; any operational deployment would need a separate review of data handling, hosting and provider requirements.
Evaluation scope & project status
Independent prototype, in development. The public examples and published evaluation use synthetic fixtures. The recorded evaluation covers clean English text reconstructed from hand-authored ground truth, not PDF parsing, OCR, scans, photos, other layouts or real operational documents. System checks support review; they do not authorise shipment or replace professional judgment. The rule implementation is not professional or regulatory validation. No production-traffic performance is claimed.
Roadmap snapshot · 2026-06-12 (in development · design only)
- in development Commodity-code lookup (AHECC): keyword matching implemented. Three API endpoints provide keyword matching over a 5,613-entry AHECC dataset, tariff lookup and free-trade-agreement (FTA) lookup. AI-assisted classification and a user interface remain unimplemented.
- design only Explain why a check flagged an issue, without changing the case.
- design only Collect shipment documents from an email conversation into a case.
Historical status record: 2026-06-12