Batch vs Real-Time Predictions: Choosing a Serving Pattern
Last updated:
Real-time is the default request and rarely the default need
Ask a stakeholder whether the churn scores should be real-time and the answer is almost always yes. It sounds better. Then ask what anyone would do differently with a score updated at 2:14pm rather than at 6am, and the room goes quiet.
The serving pattern is one of the biggest cost and complexity decisions in a machine learning project, and it is often made on vibes. It should be made on one question: when does the information needed for the prediction exist, and when does the decision happen?
The serving patterns side by side
| Pattern | How it works | Typical latency | Good for |
|---|---|---|---|
| Batch | Score all records on a schedule, write results to a table | Hours | Churn scores, lead scores, demand forecasts, stock planning |
| Near real-time | Score records within minutes of an event, from a queue or stream | Seconds to minutes | Order risk flags, alert triage, updating a score after a support call |
| Online (real-time) | An API returns a prediction per request | Milliseconds | Checkout fraud checks, search ranking, live pricing quotes |
| On-device | The model runs in the app or browser | Milliseconds, no network | Offline use, privacy-sensitive inputs, camera features |
Most business models belong in the first row. A good share of the rest are satisfied by the second. The third and fourth are for cases where the decision literally cannot wait.
Why batch prediction wins more often than people expect
- It is cheap. A job runs for twenty minutes a night. There is no server waiting around for requests.
- It is simple to debug. Every prediction is sitting in a table. You can query it, chart it and compare it with yesterday.
- Failure is gentle. If tonight's job fails, yesterday's scores are still there. A failed real-time service means no answer at all.
- Features can be heavy. Batch jobs can join ten tables and compute 90-day aggregates without anyone waiting.
- Monitoring is easier. Checking one batch of inputs against the training profile is straightforward.
A 40-person distributor scoring 12,000 customer accounts for reorder likelihood gains nothing from real-time. The sales team plans calls each morning. A nightly batch feeds the CRM and that is the whole architecture.
When you genuinely need real-time predictions
Real-time earns its cost when two things are true together: the input only exists at the moment of the request, and the answer changes what happens next within seconds.
- A payment fraud check on a basket that did not exist a minute ago
- Ranking search results for a query just typed
- A quote price that depends on the options a customer just picked
- Routing an inbound chat based on its first message
If only the first condition holds, near real-time from a queue is usually enough. If only the second holds, precompute in batch and look the answer up quickly. Plenty of systems that look real-time to the customer are a fast lookup of a score calculated overnight.
What real-time adds to the bill
Moving from batch to online serving is not a small switch. The model stays the same; nearly everything around it changes.
- An always-on service with redundancy, because an outage now blocks a customer-facing process
- A feature pipeline that can compute inputs in milliseconds, often a separate store of precomputed values
- Guarantees that live features are calculated exactly as they were in training
- Latency budgets, load testing and autoscaling
- A fallback for when the model is slow or down, such as a default decision or a rule
- Per-request logging at volume
Point three is the one that causes most production bugs. In batch, training and scoring can share the same SQL. In real-time, someone rewrites the feature logic for speed, and small differences creep in. The model then quietly performs worse live than it did in testing.
We covered the equivalent choice for language model features in batch versus real-time AI integration; the reasoning is similar, though the costs sit in different places.
A hybrid that suits many businesses
The pattern we reach for most often is batch plus a light real-time layer. Heavy features, such as a customer's twelve-month purchase history, are computed overnight. At request time a small service combines those with the few live inputs, such as the current basket, and runs a quick model.
It gives near-instant answers without rebuilding the whole data pipeline for millisecond latency. For low-traffic products, running that small layer on a serverless function is often the cheapest option, which we go into in serverless inference for low-traffic products.
How to decide
Write down, for the prediction in question: when the inputs become available, when the decision is taken, what happens if the prediction is two hours old, and what happens if the prediction service is down. If a two-hour-old answer is fine, build batch. If it is not, check whether near real-time would do before committing to online serving.
SpiderHunts starts every machine learning build in batch unless those answers force otherwise. It is easy to move a proven batch model to real-time later. Going the other way, after paying for real-time infrastructure nobody needed, is an awkward conversation.
Frequently asked questions
What is batch prediction in machine learning?
What is online inference?
Is real-time machine learning more accurate?
Can we start with batch and move to real-time later?
Being told your model needs to be real-time?
Tell us where the prediction is used and how fresh it needs to be. We will say whether a nightly batch would do the job, and what real-time would genuinely add.