What a Data Pipeline Does
Last updated:
The problem it solves
Three people produce three different revenue figures for the same month, and all three are reading a real system correctly. Finance uses invoices, sales uses the CRM, operations uses despatches. Nobody is wrong and nobody agrees.
A pipeline fixes this by defining, once, what each number means and where it comes from — then computing it the same way every time.
The deliverable is not a dashboard. It is the end of the meeting where people argue about whose number is right.
What one consists of
- Extract. Pull from each source on a schedule, incrementally where possible.
- Land raw. Store what arrived, unmodified, so you can reprocess without re-fetching.
- Transform. Clean, deduplicate, reconcile, apply the agreed definitions.
- Serve. A warehouse or database anyone can query.
- Monitor. Row counts, freshness, failed loads, values outside expected ranges.
Step two is the one that gets skipped and the one that saves you later. When a transformation turns out to be wrong in month four, having the raw history means fixing it and reprocessing rather than losing the period.
Definitions before technology
Before any code, agree what the words mean. What is an active customer? Does revenue count at order, despatch or invoice? Is a refund a negative sale or an adjustment?
These are business decisions and they take a couple of workshops. Skipping them produces a technically flawless pipeline that computes numbers nobody agrees with, which is the most expensive kind of correct.
Keeping it trustworthy
- Freshness alerts — a stale dashboard is worse than an empty one
- Row count checks against the source
- Reconciliation against a system of record, daily
- Visible last-updated timestamps on every report
- A named owner for each metric definition
Trust is the whole product. One occasion where the dashboard was wrong and nobody was told costs more credibility than six months of correct reports earn.
Right-sizing it
| Business | Sensible approach |
|---|---|
| Under 5 systems, small volumes | Scheduled scripts into a managed database |
| 5–15 systems | Managed pipeline tooling plus custom extractors |
| Large or regulated | Warehouse with proper modelling and lineage |
Most mid-sized businesses need the first row and get sold the third. A nightly job into a managed database, with monitoring, answers the actual questions for a fraction of the cost.
What it costs
A first pipeline covering two or three sources with a handful of agreed metrics is typically £10,000–£30,000, plus £100–£600 a month to run depending on volume.
The ongoing cost people forget is definition maintenance. Businesses change, and a metric definition nobody revisits becomes quietly wrong. Half a day a quarter keeps it honest.
Frequently asked questions
Do we need a data warehouse?
How often should data refresh?
Can this feed AI or forecasting later?
Who maintains it?
Need data your systems do not currently give you?
Tell us what question you are trying to answer. We will tell you whether it is a pipeline, a scraper or a report you already could be running.