A retrieval-augmented system that extracts structured data from unstructured shipping documents, with an evaluation harness that made quality measurable and defensible.
(01) The challenge
Operations staff manually keyed data from thousands of inconsistent supplier documents each week. Previous LLM prototypes produced confident but wrong extractions, so the team had lost trust in automation.
(02) Our approach
We built a retrieval and extraction pipeline with a labelled evaluation set, confidence scoring, and a human-in-the-loop fallback that routes low-confidence documents to review instead of guessing.
(03) The work
[Placeholder case study — all client details and metrics are illustrative and must be replaced with verified data before launch.]
Supplier documents arrived in every format imaginable: scans, inconsistent PDFs, and spreadsheets with different column layouts for every partner. Operations staff keyed the data by hand, which was slow, expensive, and error-prone.
An earlier LLM prototype had made things worse. It extracted plausible-looking but incorrect values with no indication of uncertainty, so the team stopped trusting it.
Built a labelled evaluation set first. Before writing any production code, we assembled a representative set of documents and correct outputs. This became the ground truth every subsequent change was measured against.
Treated extraction as retrieval plus validation. Documents are parsed into structured chunks, retrieved with hybrid search, and extracted against a strict schema. Retrieved content is treated as untrusted input to reduce prompt-injection risk.
Scored confidence explicitly. Each field carries a confidence score. Documents below threshold route to a review queue with the source region highlighted, so staff verify rather than re-key.
Measured cost and latency per document. Routing and caching keep unit costs viable, and a dashboard tracks accuracy, cost, and throughput together.
Manual data entry fell 74%. Extraction accuracy improved from 81% to 97.5% on the evaluation set, and processing time dropped from six minutes to fourteen seconds per document. Cost per document fell 68%.
The most important shift was organisational: because quality was measured, operations leadership could set an explicit accuracy threshold and let the system run against it. Trust came from evidence, not enthusiasm.
[Placeholder quote — replace with a real client testimonial before launch.] What convinced us was not the demo, it was the evaluation report. We could see exactly how accurate the system was before we trusted it with production documents.
(More work)
(Start here)
We'll map it to a measurable target and tell you honestly what it takes.
Book a call