Serverless Machine Learning Inference for Low-Traffic Products
Last updated:
An endpoint that costs more than it earns
A SaaS startup adds a lead-quality score to its product. The data scientist deploys the model to a dedicated endpoint on a managed platform, which is how the tutorial did it. The feature gets used perhaps 300 times a day. The endpoint runs 24 hours a day, every day, and the monthly bill for it quietly exceeds the revenue from the customers using the feature.
This is one of the most common and least discussed machine learning cost problems in small products. The model is fine. The serving choice is wrong for the traffic.
How serverless inference works
Serverless platforms such as AWS Lambda, Google Cloud Run or Azure Functions run your code in response to a request and shut it down afterwards. You package the model and a small prediction function; the provider handles servers and scaling. Several ML platforms also offer serverless endpoint options on the same principle.
- A request arrives and the platform starts a container if none is warm
- The container loads the model into memory, which is the slow part
- The function computes features, runs the prediction and returns the result
- The container stays warm for a while to handle more requests, then is shut down
- You pay for execution time and memory, often with a free allowance
At low traffic the bill can drop to a very small amount. At high steady traffic, per-request charges eventually overtake the cost of a server that runs all the time.
When serverless is a good fit
| Factor | Good fit | Poor fit |
|---|---|---|
| Traffic | Low, spiky or unpredictable | High and steady around the clock |
| Model size | Small to medium, loads in a second or two | Large deep learning models, hundreds of megabytes and up |
| Hardware | CPU is enough | Needs a GPU |
| Latency | An occasional slower response is acceptable | Every response must be fast, every time |
| Dependencies | Lean libraries, such as scikit-learn or a gradient-boosting library | Heavy frameworks that bloat the package |
Classic tabular models are almost ideal. A gradient-boosted tree for scoring leads, estimating a delivery time or flagging a risky order is small, CPU-friendly and quick to load.
Cold starts and how to live with them
The main trade-off is the cold start: the delay when a request arrives and no warm container exists. For a lean model it may be a second or two; for a heavy one with large libraries it can be far longer.
- Keep the package small. Strip unused libraries and choose lighter model formats where possible.
- Load the model once when the container starts, not on every request.
- Use the provider's minimum-instances or provisioned setting if a warm container is worth a small fixed cost.
- Put the call somewhere a short delay is acceptable, such as after a form submit rather than on every keystroke.
- Precompute heavy features in batch so the function does very little work.
Point five is the most effective of all. If most of the input is a customer profile computed overnight, the serverless function only has to look it up, add a few live values and run the model. That pattern also comes up in our comparison of batch and real-time predictions.
Limits that rule it out
- Package and memory limits that a large model or framework will exceed
- Execution time limits, a problem for slow models or big batch requests
- Little or no GPU support on most general serverless offerings
- Harder local debugging and less control over the runtime
- Per-request costs that stop being cheap once traffic is consistently high
Serverless is cheapest exactly when it looks least impressive: a small model, a handful of requests, and nobody waiting in a hurry.
For heavier models there are other routes: a small container service that scales to zero, a modest always-on instance, or a batch job if predictions can wait. For language models, calling a hosted API is usually cheaper than running your own at low volume. Serverless architecture explained covers the wider trade-offs beyond machine learning.
Monitoring does not go away
Serverless removes server maintenance. It does not remove the need to log predictions, check inputs and measure accuracy. Write each prediction with its model version to a database or log store, the same as any other deployment, and set alerts on error rate and latency, including cold-start latency.
Model updates are pleasantly simple: deploy a new function version with the new model file, route traffic to it, and keep the old version to roll back to.
What we would check first
When SpiderHunts reviews a model's serving costs, we look at requests per hour across a normal week, model file size and load time, latency requirements and whether any features could be precomputed. For a lot of early-stage SaaS products, that review ends with a move to serverless or batch and a noticeably smaller bill.
Occasionally it ends the other way: traffic has grown and a small always-on service is now cheaper. The point is to choose on the numbers you actually have.
Frequently asked questions
Can you run machine learning models on AWS Lambda?
What is a cold start in serverless inference?
Is serverless cheaper than a dedicated model endpoint?
Does serverless work for deep learning models?
Paying for a model endpoint that mostly sits idle?
Tell us your request volume and model size. We will estimate whether serverless inference would cut the bill, and flag anything that would make it a poor fit.