Case study — Verdict

Verdict is an independent prototype for reviewing shipment paperwork. I designed the workflow, system boundaries and evaluation, using AI-assisted implementation. It brings conflicting document fields and their supporting evidence into one review flow.

The problem

Shipment information arrives across air waybills, invoices, packing lists and declarations. A reviewer has to spot missing documents and conflicting quantities, weights or parties, then prepare a clear handover. I built a workflow that brings those comparisons together and keeps the supporting details visible.

Designed around the review, not just the model

  • Access and change boundaries

    Authentication is followed by case-owner checks. Completed pre-checks must be reopened before reviewed inputs can change.

  • Failures stay local to a file

    Bounded retries and per-file failure handling keep one extraction failure from discarding the whole batch.

  • Findings lead back to their basis

    Findings identify the rule, documents, fields and values behind a conflict. A person verifies them and reviews follow-up drafts before acting.

My central design decision

I separate model extraction from rule-based decisions: the model proposes fields; deterministic rules assign the system’s review status and findings. Reviewers verify the sources, correct values and decide the next action.

Architecture

  1. Input

    Documents

    Bring documents together while retaining field sources.

  2. Model-assisted extraction

    Propose fields

    The model extracts fields. Extracted values can still be wrong.

  3. Deterministic core

    Check structure

    Validate shape and types, not whether the content is true.

  4. Deterministic core

    Apply written checks

    Written checks assign system status and identify findings.

  5. Human judgment

    Human review

    A person checks evidence and gaps, then decides what happens next.

Assisted drafting · outside the deterministic decision core

Templates and model assistance prepare follow-ups for review before sending or acting.

Traceable records

  • audit log
  • field corrections
  • document versions

Records make inputs and changes inspectable; they do not establish truth.

Model output crosses a structure-validation boundary before written checks assign review status. A person verifies the findings and decides what happens next.
Explore implementation details

Schema validation checks structure, not whether a field is true. Compliance-review reports use SHA-256 to identify a snapshot of rule metadata. That snapshot records rule descriptions, not the executable conditions or proof that a result is correct.

Document classification
heuristic classifier → LLM fallback
Model extraction
JSON-mode extraction
Structure validation
Zod schema validation
Deterministic checks
(57 across compliance review and shipment pre-check)
Rule output
review status + findings

What decides what

TaskDeterministic logicLanguage model (LLM)Reviewer
Assign a review statusAssign pass, review or block—Verify findings and decide next steps
Extract document fieldsCheck structure and typesPropose field valuesCheck and correct values
Identify document typesMatch keywords firstAssist when confidence is low—
Prepare follow-up emailsApply templatesAssist with draftingReview before use
Calculate a rule-based scoreCalculate the score—Interpret alongside findings; not a probability

Recorded verification

The prototype documents 9 document types and 57 checks across compliance review and shipment pre-check. Project records list approximately 1,000 test cases and 8 adversarial test rounds. CI checks tests, types, build and lint; these checks do not cover every real-world situation.

A separate June 2026 evaluation made one recorded pass over clean English text reconstructed from 85 synthetic documents in 17 sets. Extracted values matched the expected values in 96.3% of 511 fields after normalisation and numeric tolerance; allowing substring containment raised agreement to 99.0%. PDF parsing and OCR were not evaluated.

Data boundary & provider choice

Model calls sit behind a provider interface, separate from the decision engine. DeepSeek and MiniMax are implemented and selected by configuration, with bounded retries within a provider—not automatic cross-provider failover. A new backend requires an adapter and validation. The public examples use synthetic data; any operational deployment would need a separate review of data handling, hosting and provider requirements.

Evaluation scope & project status

Independent prototype, in development. The public examples and published evaluation use synthetic fixtures. The recorded evaluation covers clean English text reconstructed from hand-authored ground truth, not PDF parsing, OCR, scans, photos, other layouts or real operational documents. System checks support review; they do not authorise shipment or replace professional judgment. The rule implementation is not professional or regulatory validation. No production-traffic performance is claimed.

Roadmap snapshot · 2026-06-12 (in development · design only)

  • in development Commodity-code lookup (AHECC): keyword matching implemented. Three API endpoints provide keyword matching over a 5,613-entry AHECC dataset, tariff lookup and free-trade-agreement (FTA) lookup. AI-assisted classification and a user interface remain unimplemented.
  • design only Explain why a check flagged an issue, without changing the case.
  • design only Collect shipment documents from an email conversation into a case.

Historical status record: 2026-06-12

← Back to the site