Data Pipelines That Keep Machine Learning Models Fed

The model is the easy part to keep running

A trained model is a file. It does not wear out. What wears out is the flow of data into it. A column gets renamed in the CRM, an overnight export fails, a new product category appears, and the model carries on producing numbers that look perfectly normal and are quietly wrong.

In our experience, most models that disappoint after launch were let down by their data pipelines, not their algorithms. The pipeline is the part that needs engineering attention for the whole life of the model, which is why it deserves more planning than it usually gets.

Two pipelines, not one

Every production model needs two related flows of data.

• The training pipeline pulls historical data, cleans it, calculates features as they were at each past date, attaches outcomes and produces a versioned training dataset. It runs whenever the model is retrained, perhaps monthly.
• The prediction pipeline pulls the latest data, cleans it the same way, calculates current features, runs the model and writes predictions somewhere useful, such as the CRM, a dashboard or an application. It runs as often as predictions are needed.

The critical rule is that both use exactly the same cleaning and feature code. When they drift apart, the model sees different numbers in production from those it learned on, a problem known as training-serving skew. Sharing code, or sharing feature tables as described in our post on feature stores, is the defence.

How fresh does the data need to be?

Real-time pipelines are expensive to build and run. Most business predictions do not need them. Match freshness to how quickly the decision is actually made.

If someone proposes streaming for a model whose output is read in a Monday meeting, question it. Moving from daily to real time can easily multiply build and running costs. When real time genuinely is needed, our post on real-time data streaming with Kafka covers what is involved.

Checks that catch silent failures

A data pipeline for machine learning needs checks at each stage that stop the run and alert someone, rather than passing bad data along.

1. Arrival. Did every expected source deliver, and is the data from the right date?
2. Schema. Are all expected columns present, with the right types?
3. Volume. Is the row count within a sensible range compared with recent runs?
4. Values. No impossible values such as negative quantities or dates in the future; categories from the known list.
5. Distribution. Have key features shifted sharply from what the model was trained on?
6. Predictions. Is the share of high-risk scores or the average forecast within a normal range?

The last two catch the subtle problems. If the model suddenly flags 40% of customers as likely to churn when it normally flags 8%, something upstream has almost certainly changed. General pipeline reliability is covered in building a data pipeline that does not break.

Deciding what happens when it breaks

Pipelines will fail. The design question is what the business sees when they do. Our default is to keep the most recent good predictions in place, mark them with their date, and alert the owner. A sales team working from yesterday's lead scores is fine; one working from scores calculated on half a dataset is not.

• Write predictions to a new table or version and only switch over once all checks pass
• Show the date predictions were produced wherever they are used
• Set a maximum age after which stale predictions are hidden or flagged clearly
• Keep a simple fallback, such as a rule-based score, for longer outages
• Send alerts to a named person, not a shared inbox nobody reads

Tools, kept proportionate

For one or two batch models, a scheduled job running Python or SQL, with logging and alerts, is entirely adequate. As the number of pipelines grows, an orchestrator such as Airflow, Dagster or Prefect helps manage dependencies, retries and history. Tools such as dbt keep SQL transformations versioned and tested. Monitoring tools can track feature drift and prediction distributions.

Resist building the full stack for your first model. Start simple, make sure the checks and fallback exist, and add tooling when the pain of managing several pipelines by hand becomes real.

What it costs to run and look after

Beyond the initial build, budget for ongoing care. Source systems change, new categories appear and upstream teams forget to mention it. A typical small setup needs a few hours a month of attention in quiet periods and more when a connected system is upgraded.

When SpiderHunts hands over a model, the pipelines, checks, alerts and fallback behaviour are part of the delivery, along with a short runbook describing what to do when each alert fires. We also offer ongoing support for teams without an in-house data engineer. If your models are fed by fragile scripts, our automation service and machine learning team can put something dependable in place.

Frequently asked questions