The input will be wrong
Pipelines fail in production because real data contains things the sample did not: missing fields, unexpected encodings, dates in a different format, and records that break an assumption nobody wrote down.
A pipeline that stops on the first bad record will stop most nights. One that quarantines bad records and processes the rest keeps working while you investigate.
Design principles
- Validate on entry and quarantine what fails, with the reason
- Make each stage restartable from its input rather than from the beginning
- Store intermediate output, so a later failure does not require redoing earlier work
- Be idempotent — rerunning must not duplicate
- Reconcile — count in, count out, count quarantined
Reconciliation is the safety net
Every run should report how many records entered, how many completed, how many were quarantined and how many were rejected. Those numbers should add up.
- An unexplained gap means silent data loss
- A quarantine count that jumps means the input changed
- A completed count of zero means something upstream broke
- All three should alert
Handle the awkward realities
| Reality | Handling |
|---|---|
| Encodings vary | Detect and normalise on entry |
| Dates in several formats | Parse explicitly, reject ambiguity |
| Duplicate source records | Deduplicate on a defined key |
| Late-arriving data | Design for reprocessing a period |
| Source unavailable | Retry, then alert — do not silently skip |
Alert on absence, not just errors
A pipeline that processes nothing looks identical to a pipeline with nothing to process. Only a completion signal with an expected volume distinguishes them.
That is the alert that catches the failures which otherwise run for weeks.