Where the cost actually goes
Scoring a single record involves far more than the model computation. There is a network request, authentication, loading or holding the model in memory, fetching features, serialising the result and logging it.
That overhead is largely fixed per request. Scoring a thousand records in one operation pays it once rather than a thousand times, which is why the cost difference is usually large rather than marginal.
When batching is available to you
| Situation | Batchable? |
|---|---|
| Scores shown on a dashboard | Yes - refresh on a schedule |
| Overnight operational process | Yes - naturally batch |
| Customer scores for a campaign | Yes - computed before the send |
| Prediction depends on the current basket | No - genuinely live |
| Fraud check during payment | No - must be immediate |
| Recommendation on a product page | Often precomputable per customer and product |
That last row deserves attention. A great deal of what is built as live scoring could be precomputed, because the set of things you might be asked about is bounded - your customers and your products.
Precompute and serve from a table
The pattern that gets real-time responsiveness at batch cost: compute predictions on a schedule, write them to a fast lookup store, and serve from there.
- Identify the bounded set - all active customers, all products in stock.
- Score them on a schedule matched to how fast the inputs change.
- Write results with a timestamp to a store built for reads.
- Serve by lookup, which is fast and cheap.
- Decide what happens for something not in the store - a new customer - and have a default ready.
The final step is where this design usually fails in production. New entities appear constantly, and a lookup that returns nothing needs a defined fallback rather than an error.
How stale is too stale
The freshness question is answered by what the prediction depends on. A churn score built on months of behaviour does not change meaningfully in an hour. A score depending on this session's clicks does.
Where the answer is mixed, a hybrid works: a precomputed base score adjusted at request time by a small live component. That keeps most of the cost saving while reflecting what just happened.
Do not batch everything blindly
Batching has costs too. Precomputing scores for every customer when only a small share are ever looked at wastes compute, particularly with an expensive model.
Where the set is large and access is sparse, scoring on demand with caching is usually better - compute once when first requested, reuse for a defined period. That gets most of the benefit without precomputing for people nobody asks about.
Most real-time scoring is answering a question that could have been answered last night.