Data Pipelines That Fail Loudly Instead of Quietly

Green does not mean correct

Most pipeline monitoring reports whether the job ran. That catches crashes, which are the easy failures. It misses the expensive ones: a source that changed format so a field now parses as null, a filter that silently excludes half the records, a join that duplicates rows.

Those failures produce a green tick and wrong numbers, and they are typically discovered when someone questions a report weeks later — by which time decisions have been made on bad data.

Four checks that catch most of it

1. Row count against expectation. If yesterday brought 40,000 records and today brings 400 or 400,000, something changed. Alert on deviation from a rolling band, not on zero.
2. Field completeness. Track the null rate per important column. A jump from 2% to 60% is a format change, and it is invisible in a job-status dashboard.
3. Value distribution. Prices suddenly ten times higher usually means a currency or units change, not a market event.
4. Freshness. When was the newest record created? Stale data that keeps loading is a common and silent failure.

Keep the raw data

Store the source response before transformation, cheaply and durably. When you discover a parsing error three weeks later, this is what lets you reprocess history instead of losing it.

Object storage for raw payloads costs very little and has rescued more projects than any other single practice we apply. The one time you need it, it saves weeks.

Make failures partial, not total

One malformed record should not stop the run. Route bad records to a quarantine table with the error and the original payload, process everything else, and report the quarantine count.

Then actually look at the quarantine. A growing pile nobody reviews is the same as silent failure with extra steps — so put the count on the same alert as everything else.

Make it idempotent and re-runnable

You will need to re-run a day. Design so that running the same input twice produces the same result rather than duplicates — natural keys, upserts, or a run identifier you can delete cleanly.

• Every record traceable to the run that produced it
• Re-running a date range replaces rather than appends
• Backfilling a fix is a normal operation, not an incident
• Schema changes are versioned so old data remains readable

Alert on what needs a human

Every alert should be actionable. If nobody would do anything at 9am on receiving it, it should be a dashboard line, not an alert. Teams that alert on everything stop reading alerts within a fortnight.

A reasonable split: page for freshness and total failure, email for distribution anomalies and quarantine growth, dashboard for everything else.

Frequently asked questions