Handwriting Recognition for Forms and Delivery Notes

The clipboard is not going anywhere

A haulage firm runs 60 drivers. Every drop ends with a customer scrawling a signature and sometimes a note on a paper delivery note: 'two pallets damaged', 'left with neighbour at 14'. The notes come back in a crumpled stack, and an administrator spends each morning keying quantities, exceptions and proof of delivery into the transport system.

Paper persists for good reasons. It works without signal, customers will sign it, and it survives rain better than a cracked tablet. Rather than fight that, a lot of businesses now photograph or scan the paper and let a model read it.

Handwriting recognition has improved a great deal. It is still the one area of document extraction where you should expect to keep a person in the loop for a while.

What handwriting recognition reads well, and what it does not

The difference between a tidy inspection form and a delivery note written on a van bonnet in the rain is larger than the difference between any two models.

Traditional OCR versus modern vision models

Older OCR engines were trained on printed text and struggled badly with handwriting. Specialist handwriting engines improved on that for known form layouts. General multimodal models now read handwriting in context: they understand that a scribble in the 'Qty' box is probably a number, and that 'lft w/ neighbr' means left with neighbour.

That contextual reading is a double-edged sword. It fixes a lot of errors and occasionally invents a plausible answer where the ink is genuinely illegible. A traditional engine returns garbage you can see is wrong. A language-aware model can return something tidy that is wrong. Our comparison of OCR and LLM document extraction goes into that trade-off for printed documents; with handwriting it is sharper still.

In practice we often use both: a vision model to extract fields, plus validation rules to catch anything that does not fit.

Validation does more work than the model

The most effective accuracy improvements come from checking extracted values against things you already know.

• Order numbers must match an open order in the transport or ERP system
• Quantities must not exceed what was dispatched on that order
• Dates must fall within the delivery window
• Customer names are matched fuzzily against the drop address on the manifest
• Totals on timesheets must equal the sum of the rows
• Free-text exceptions are classified into a short list, such as damaged, short, refused or left safe

A field that the model reads with high confidence and that passes validation can post automatically. A field that fails either test goes to a person with the image crop shown next to the suggested value, so correcting it takes seconds rather than re-reading the whole page.

Capture quality matters more than you would think

Most accuracy problems begin at the moment someone photographs the page. Crumpled paper on a dashboard, a thumb over the quantity column, a shadow across half the form.

1. Use a capture app with edge detection and automatic flattening rather than the phone's camera roll
2. Reject blurry or cut-off images at the point of capture, while the driver is still standing there
3. Redesign the paper form: boxed fields, one character per box for numbers, and space for notes
4. Print a QR code with the order number on every note so the key field never needs reading

That last one is our favourite. The single most important field on a delivery note is the one you can print in machine-readable form before it ever leaves the depot.

When you should go digital instead

If drivers or engineers already carry a phone or rugged device, and customers are happy to sign on glass, a well-designed mobile form is more reliable than reading paper afterwards. Typed data does not need recognising. The honest reason to read handwriting is that paper is staying for customer, signal or habit reasons, not that nobody has tried digital.

For the in-between cases, a hybrid works: digital for structured fields the driver controls, and a photographed paper note for the customer's signature and comments. The document automation pipeline then handles both in one place.

What a project looks like

At SpiderHunts we start with a sample of a few hundred real scanned documents, including the worst ones, hand-labelled with the correct values. We measure accuracy per field, not per document, because a note where the order number is right and the free text is wrong is still mostly useful.

Typical findings: numbers and tick boxes clear automatically at high rates, free-text exceptions need review more often, and the review screen design decides how much time is actually saved. Illustratively, if a team of two keys 400 notes a day and the system clears three quarters of fields automatically, the job becomes one person checking exceptions for part of the morning.

Frequently asked questions