Moving and Transforming Data Reliably

The input will be wrong

Pipelines fail in production because real data contains things the sample did not: missing fields, unexpected encodings, dates in a different format, and records that break an assumption nobody wrote down.

A pipeline that stops on the first bad record will stop most nights. One that quarantines bad records and processes the rest keeps working while you investigate.

Design principles

1. Validate on entry and quarantine what fails, with the reason
2. Make each stage restartable from its input rather than from the beginning
3. Store intermediate output, so a later failure does not require redoing earlier work
4. Be idempotent — rerunning must not duplicate
5. Reconcile — count in, count out, count quarantined

Reconciliation is the safety net

Every run should report how many records entered, how many completed, how many were quarantined and how many were rejected. Those numbers should add up.

• An unexplained gap means silent data loss
• A quarantine count that jumps means the input changed
• A completed count of zero means something upstream broke
• All three should alert

Handle the awkward realities

Alert on absence, not just errors

A pipeline that processes nothing looks identical to a pipeline with nothing to process. Only a completion signal with an expected volume distinguishes them.

That is the alert that catches the failures which otherwise run for weeks.

Frequently asked questions