What a Machine Learning Model Really Costs to Run
Last updated:
The business case usually stops at launch
Most proposals for a machine learning model include a build cost and a projected benefit. Few include a line for keeping it working in year two. That gap is why so many models are quietly switched off eighteen months later, still technically running, no longer trusted.
The running cost of a model is not dramatic. It is also not zero, and it rarely sits where people expect. For the classic prediction models most SMEs use, the servers are the cheap part.
Where the money actually goes
| Cost area | What it covers | Size for a typical SME model |
|---|---|---|
| Compute for predictions | Running the batch job or prediction service | Usually small, often within existing hosting |
| Compute for training | Retraining runs and experiments | Small for tabular models, significant for deep learning |
| Data pipelines | Keeping source extracts, joins and cleaning working | Moderate, and spiky when source systems change |
| Monitoring and reporting | Prediction logs, dashboards, alert review | Small to set up, steady time to review |
| Retraining and validation | Running, checking and approving new versions | A few days per cycle |
| Incident fixes | Chasing bad predictions and data faults | Unpredictable, budget a contingency |
| Licences and platforms | Managed ML platforms, monitoring tools, data warehouse | Anything from nothing to a noticeable monthly fee |
Everything from the third row down is mostly people's time. That is the part to be honest about.
An illustrative yearly budget
Take a made-up but typical case: a 70-person online retailer running a nightly model that predicts which customers are likely to lapse, feeding an email tool. Around 50,000 customers are scored each night. The figures below are an example of how the effort breaks down, not a quote.
- Hosting the nightly job and storing prediction logs: a small fraction of an existing cloud bill
- Four quarterly retrains, each with checks and a comparison report: roughly two to three days of engineering each
- Monthly review of the accuracy report by the marketing lead and an engineer: an hour or two each month
- Fixing breakages when the ecommerce platform or email tool changes its data: typically a few days a year, unevenly spread
- An annual review of whether the model still earns its place: a day
None of those items is large on its own, which is exactly why they get left out of budgets. Each one feels like something that will be fitted in around other work. In practice, work that is fitted in around other work gets skipped in busy months, and busy months are when data sources change.
Add that up and it is somewhere around three to four weeks of skilled time a year, plus modest hosting. If the model helps win back even a small share of lapsing customers, that is easy to justify. If the benefit is marginal, the running cost is what should prompt the question of whether to keep it.
When compute does become the big number
Compute starts to dominate in a few specific situations, and it is worth knowing whether you are in one of them before signing off an architecture.
- Always-on real-time services with redundancy, paid for around the clock even when traffic is low
- Deep learning models for images, audio or long text, especially if they need GPUs to serve
- Frequent retraining of large models on big datasets
- Language model calls at volume, where each prediction has a per-token price
- Managed platform endpoints left running for models that are used a few times a day
The last one catches people constantly. A dedicated endpoint for a model that serves forty requests a day can cost more per year than the model's entire build. Serverless inference or plain batch scoring usually fixes it. For GPU-heavy work, our note on GPU cloud costs for inference goes into the options.
Costs that do not show up on any invoice
Some of the most expensive parts of running a model are never billed.
- The cost of wrong predictions that nobody noticed, which is why monitoring pays for itself
- Staff time spent working around a model they no longer trust
- The key-person risk when one contractor or employee understands the pipeline
- Compliance work if the model affects decisions about people, including documentation obligations under the EU AI Act for some uses
The cheapest model to run is one that is monitored. The most expensive is one that is still running long after it stopped being right.
How to keep running costs down
- Prefer batch serving unless real-time is genuinely needed
- Choose simpler models when accuracy is close; a gradient-boosted tree is cheaper to run and explain than a neural network
- Reuse existing infrastructure, such as your current database and scheduler, before adding platforms
- Automate the retraining run but keep reviews short and structured
- Retire models that no longer earn their keep, which almost nobody does
When SpiderHunts quotes a machine learning project, we include a year-two running estimate alongside the build price, split into hosting and people time. Clients have occasionally decided not to proceed once they saw it. That is a better outcome than building something they could not afford to maintain.
Frequently asked questions
How much does it cost to host a machine learning model?
What are the ongoing costs of machine learning?
Is it cheaper to use a managed ML platform?
When should we switch a model off?
Budgeting for a model and unsure what to allow?
Describe the model and how often it runs. We will give you a rough yearly running cost, including the parts that usually get left out of the business case.