Think Build Implement Repeat
London, UK +44 7367 067226
WhatsApp FOLLOW f in X
  1. Home
  2. Blog
  3. Batching Predictions to Reduce Cost
Cloud & DevOps

Batching Predictions to Reduce Cost

Scoring records individually is the most expensive way to run a model. When batching works, and how to keep results fresh enough.

Updated 2 min readBy SpiderHunts Technologies

Free estimateNo obligation

Get a free estimate

Tell us what you need. A senior engineer reads every enquiry.

Takes under a minute. We never share your details.

  • Free consultation
  • No commitment
  • NDA on request

Prefer to talk? Book a free 30-minute call →

Quick answer — TL;DR

Per-record scoring carries fixed overhead every time. Batching amortises it and frequently reduces cost by an order of magnitude. The question is whether the prediction needs to reflect something that happened in the last few minutes - often it does not.

Where the cost actually goes

Scoring a single record involves far more than the model computation. There is a network request, authentication, loading or holding the model in memory, fetching features, serialising the result and logging it.

That overhead is largely fixed per request. Scoring a thousand records in one operation pays it once rather than a thousand times, which is why the cost difference is usually large rather than marginal.

When batching is available to you

SituationBatchable?
Scores shown on a dashboardYes - refresh on a schedule
Overnight operational processYes - naturally batch
Customer scores for a campaignYes - computed before the send
Prediction depends on the current basketNo - genuinely live
Fraud check during paymentNo - must be immediate
Recommendation on a product pageOften precomputable per customer and product

That last row deserves attention. A great deal of what is built as live scoring could be precomputed, because the set of things you might be asked about is bounded - your customers and your products.

Precompute and serve from a table

The pattern that gets real-time responsiveness at batch cost: compute predictions on a schedule, write them to a fast lookup store, and serve from there.

  1. Identify the bounded set - all active customers, all products in stock.
  2. Score them on a schedule matched to how fast the inputs change.
  3. Write results with a timestamp to a store built for reads.
  4. Serve by lookup, which is fast and cheap.
  5. Decide what happens for something not in the store - a new customer - and have a default ready.

The final step is where this design usually fails in production. New entities appear constantly, and a lookup that returns nothing needs a defined fallback rather than an error.

How stale is too stale

The freshness question is answered by what the prediction depends on. A churn score built on months of behaviour does not change meaningfully in an hour. A score depending on this session's clicks does.

Where the answer is mixed, a hybrid works: a precomputed base score adjusted at request time by a small live component. That keeps most of the cost saving while reflecting what just happened.

Do not batch everything blindly

Batching has costs too. Precomputing scores for every customer when only a small share are ever looked at wastes compute, particularly with an expensive model.

Where the set is large and access is sparse, scoring on demand with caching is usually better - compute once when first requested, reuse for a defined period. That gets most of the benefit without precomputing for people nobody asks about.

Most real-time scoring is answering a question that could have been answered last night.

FAQ

Frequently asked questions

The questions readers ask us after this guide.

Still have a question?

Ask us directly — a senior engineer will get back to you.

Ask about your project

How much cheaper is batching?

It varies with the model and infrastructure, but the difference is typically large rather than marginal because the fixed per-request overhead dominates for small models.

What batch frequency should we use?

Match it to how fast the inputs meaningfully change. Daily suits many customer-level scores; hourly or faster suits operational ones.

Is caching the same as batching?

Related but different. Caching stores results after they are requested; batching computes them in advance. Both reduce repeated computation.

Can we mix batch and real-time?

Yes, and it is often the best design - a precomputed base with a small live adjustment.

Keep reading

More on Cloud & DevOps

Cloud & DevOps

CI/CD for Machine Learning Projects

Software pipelines test code. Model pipelines must also test data and behaviour. What to add, and which gates should stop a release.

Start here

Want machine learning project details from us?

Tell us what you are trying to predict and roughly what data you hold. We will come back with an honest view on whether machine learning is the right tool, what the work would involve and a realistic cost range. If a spreadsheet would do the job, we will say so.

  1. You tell us what you needTwo minutes on the form, or a message on WhatsApp.
  2. A senior engineer reviews itAnd comes back with questions, a realistic range and an honest view on fit.
  3. Free 30-minute scoping callWe talk through scope, options and a realistic estimate — with no obligation.
Free estimateNo obligation

Talk to someone who builds this

Send a short brief and we will come back with an honest view and a realistic range.

Takes under a minute. We never share your details.

  • Free consultation
  • No commitment
  • NDA on request

Prefer to talk? Book a free 30-minute call →