Real-time is a cost decision
Serving predictions on demand within a page load is substantially more expensive to build and run than computing them overnight. It needs an always-available service, low-latency access to features, monitoring and an answer for what happens when it is slow.
So it is worth asking, before committing, whether the requirement is real. A surprising share of 'real-time' specifications turn out to mean 'reasonably current', which a scheduled refresh satisfies at a fraction of the cost.
Work backwards from the experience
| Where the prediction appears | Realistic budget | Implication |
|---|---|---|
| Inside a page render | Tens of milliseconds | Genuine real-time; feature access is the constraint |
| After a user action, with a spinner | A few hundred milliseconds | Real-time, more forgiving |
| In a back-office queue | Seconds | Near-real-time; simpler architecture |
| On a report or dashboard | Minutes to hours | Batch is fine |
| Driving a daily process | Overnight | Batch, and far cheaper |
Placing the requirement honestly in this table is usually a five-minute conversation that saves a great deal of engineering.
The model is rarely the slow part
For most business models, inference itself takes a few milliseconds. The time goes on everything around it.
- Fetching features from several systems, sometimes sequentially
- Computing aggregates on the fly - 'orders in the last 90 days' over a large table
- Network hops between services
- Cold starts where the service scales from zero
- Serialisation, authentication and logging
This is why feature access dominates the design. Precomputing aggregates and storing them ready to read converts the expensive part into a lookup, and is usually the single biggest latency improvement available.
Precomputing more than you expect
Where the set of things you might be asked about is bounded - your customers, your products, your sites - you can compute predictions in advance and serve them from a table.
This gives real-time response times with batch economics. It fails only where the prediction depends on something known just before the request, such as the contents of the current basket, and even then a hybrid works: a precomputed base score adjusted by a small live component.
Specify the tail, not the average
Average latency hides the problem. If the average is 40 milliseconds and the slowest one per cent takes four seconds, those slow requests are concentrated in exactly the sessions you care about - complex accounts with more data.
Set the target on a high percentile and monitor it there. And decide in advance what happens when the budget is exceeded: serve a default, skip the personalisation, or make the user wait. That decision belongs in the design, not in an incident.
Most real-time requirements are a habit. Ask what actually breaks if the number is an hour old.