Detecting Duplicate Invoices and Payments Automatically

Why the built-in check is not enough

Most accounting and ERP systems will warn you if the same supplier sends the same invoice number twice. That catches the easiest case and almost nothing else.

Real duplicates look slightly different. The supplier emails a PDF and then posts a paper copy, which gets keyed with INV-10423 one time and 10423 the next. A statement reminder gets processed as a new invoice. The same supplier exists twice in the system after a bank detail change. Someone enters 1,250.00 and someone else enters 1,205.00. None of those trip an exact-match rule.

The common patterns behind duplicate payments

Invoice capture tools that read PDFs automatically reduce keying errors, which helps. They also make it very fast to process the same document twice, which does not. Our invoice processing automation guide covers capture; duplicate detection should sit right after it.

How automated duplicate detection works

The approach has two stages: find candidates cheaply, then score them carefully.

1. Normalise. Strip prefixes and punctuation from invoice references, standardise supplier names, convert currencies, and match supplier bank details to spot duplicate supplier records.
2. Generate candidate pairs. For each new invoice, find existing invoices within a window, perhaps 90 days, with a similar amount, similar reference or same bank account.
3. Score each pair. Combine similarity on reference, amount, date, supplier, line items and PDF content into a likelihood of being a duplicate.
4. Route for review. High scores are held from the payment run pending a human check; lower scores appear in a weekly report.

The scoring can start as sensible rules: same amount, reference edit distance of one or two, within 30 days. Once reviewers have confirmed or rejected a few hundred candidate pairs, a classifier trained on those decisions ranks new candidates better than hand-tuned rules. That is the right order: rules first, model once there are labels.

Recovering past duplicates first

Before building prevention, run the same detection across the last two or three years of paid invoices. This is a one-off exercise and it tells you how big the problem actually is.

Take a manufacturer processing 25,000 supplier invoices a year. A historical sweep will usually find a list of likely duplicates, most of which turn out to be legitimate on inspection: recurring monthly charges for the same amount, instalments, or genuine repeat orders. The remainder are worth chasing with suppliers for credit. Whether that remainder is large or trivial varies enormously between businesses, which is exactly why you run the sweep before deciding to invest in anything ongoing.

Most suppliers will refund an overpayment when asked with evidence. Very few will volunteer it.

Keeping false positives manageable

Recurring charges are the main source of noise. Rent, subscriptions and maintenance contracts produce identical amounts every month by design.

• Tag recurring suppliers and compare invoices against the expected billing cycle instead
• Use line-item and period descriptions, not just header totals
• Let reviewers mark a pair as not a duplicate so it never resurfaces
• Hold only the top-scoring candidates from payment; report the rest
• Review the hold rate monthly so the AP team does not start rubber-stamping releases

A good target is a hold list short enough that someone checks every item properly before the run. If the list is long enough that people start releasing items in bulk, the check has become theatre, and the threshold needs raising.

When you do not need a custom system

If your volumes are low, a monthly export and a well-built spreadsheet check, looking for same amount within 60 days per supplier, will catch a good share of duplicates. Some AP automation platforms and audit tools also include duplicate detection that is worth switching on before building anything.

A custom build makes sense when you run several entities or ledgers, have high invoice volumes, receive invoices through several channels, or already have an invoice capture pipeline that can call a scoring service before posting.

How SpiderHunts would build it

At SpiderHunts we start with the historical sweep, because it is cheap, it produces real evidence and it often pays for the rest. If the findings justify ongoing prevention, we add a scoring step between invoice capture and approval, connected to the ledger through its API, with a simple review queue for held items.

The matching and scoring is a good fit for our automation work, with the classifier added once there are reviewed pairs to learn from. For payments diverted to fraudulent bank details, which is a related but different problem, see transaction anomaly detection for finance teams.

Frequently asked questions