Turning Files Into Structured Data

Know what you are dealing with

The first question in any document project is what proportion of the input falls into each category. That determines the achievable accuracy more than anything else.

Validation matters more than extraction

Extraction alone might reach ninety per cent on mixed documents. Extraction with validation reaches the same accuracy and knows which ten per cent it got wrong, which is what makes it usable.

• Do the line items sum to the stated total?
• Does the supplier exist in your records?
• Is the date plausible?
• Is this document a duplicate?
• Are required fields present?

Route the uncertain to a person

Anything that fails validation, or where extraction confidence is low, should reach a review queue rather than being written as though it were certain.

That review interface determines whether the system saves time. If checking takes longer than keying, people stop using it.

Spreadsheets are messier than they look

1. Merged cells that break row alignment
2. Several tables on one sheet
3. Headers that repeat partway down
4. Numbers stored as text
5. Formulas whose values matter, not the formulas

Each is common and each needs handling explicitly rather than assuming a clean rectangular table.

Preserve the source

Always keep the original document alongside the extracted data, linked. When a value is questioned six months later, the answer is in the source.

It is also what makes the audit conversation short, and it costs almost nothing in storage.

Frequently asked questions