Think Build Implement Repeat
London, UK +44 7367 067226
WhatsApp FOLLOW f in X
Python & Django

Finding the Problems Before They Cost You

Last updated:

Data degrades continuously

Duplicate customers accumulate, fields go unfilled, values drift from their permitted set, and relationships break. None of it announces itself.

Businesses discover their data quality problems during an audit, a migration or a reporting dispute — all of which are expensive moments to find out.

What to check automatically

  1. Duplicates by every plausible key, including fuzzy matching on names
  2. Missing values in fields that should always be populated
  3. Impossible values — negative quantities, future dates of birth, amounts outside plausible ranges
  4. Referential gaps — records pointing at things that no longer exist
  5. Format inconsistency — the same thing written several ways

Run them on a schedule

  • Daily, on the records created or changed that day
  • Weekly, across the whole dataset for slower-moving problems
  • Report as a count and a list, not as an alert per record
  • Track the trend, because a rising count means something changed

Fix causes, not instances

A recurring data problem is a process problem. Duplicate customers accumulate because the creation process does not check; missing fields accumulate because they are not required.

SymptomUsual cause
Duplicate customersNo check at creation
Missing fieldsNot required, or required later
Inconsistent formatsFree text where a list belongs
Orphaned recordsDeletion without cascade
Impossible valuesNo validation at entry

It pays for itself before a migration

Any future system change — a migration, an integration, a new reporting layer — is made considerably harder by poor data quality. Fixing it continuously is far cheaper than fixing it under project pressure.

It also means your reporting is trustworthy in the meantime, which is worth something on its own.

Frequently asked questions

How do we find duplicates?

Exact matching on identifiers, then fuzzy matching on names and addresses. The second finds most of them and needs review.

Should the checks fix things automatically?

Report rather than fix, except for entirely unambiguous cases. Automatic merging of records that turn out to be different is difficult to undo.

How long does this take to build?

A few days for a solid set of checks on a typical business dataset.

Who acts on the reports?

Whoever owns the data. Reports nobody acts on are as useless as no reports.

Keep reading

Data nobody quite trusts?

Automated checks find where the problems are. Happy to build a set for your records.

Book a free 30-minute call Get a project estimate WhatsApp us

Related services

What we build for problems like this one

Custom Software DevelopmentWeb DevelopmentMachine Learning