The reasons that actually hold
- Contractual or regulatory restriction - a client contract or regime that prohibits the data leaving defined infrastructure.
- Latency at a physical location - a production line or vehicle where a round trip is too slow or connectivity is unreliable.
- Sustained heavy compute - continuous high utilisation, where owned hardware can be cheaper over a few years.
- Data volume that is impractical to move - where egress or transfer time dominates.
Generalised security concern is usually not on this list. Major cloud providers offer security most organisations cannot match internally, and an on-premise server maintained by a stretched IT team is not obviously safer.
What it really costs
Hardware is the visible cost and rarely the largest. The recurring costs are people and the things that stop when nobody is watching.
| Cost | Often overlooked |
|---|---|
| Hardware and refresh | Depreciation over three to five years |
| Power, cooling, space | Meaningful for GPU workloads |
| Staff time | Patching, drivers, upgrades, on-call |
| Redundancy | A second machine, or accepted downtime |
| Backup and recovery | Tested restores, not just backups |
| Idle capacity | Sized for peak, used at average |
That last row is the structural difference. Cloud lets you rent peak capacity briefly; owned hardware sits at peak size permanently. For bursty training workloads that gap is large.
The hybrid that usually fits
The requirement is rarely 'everything on our hardware'. It is usually one specific constraint, and a split satisfies it.
- Train in the cloud on anonymised or aggregated data; serve on-premise where the sensitive data lives.
- Train on-premise where the sensitive training data must stay; serve in the cloud on non-sensitive inputs.
- Run inference at the edge for latency, with training and monitoring centralised.
- Keep the data on-premise and bring compute to it within your own network.
Writing down which specific constraint applies usually reveals which split works, and avoids paying for a full on-premise platform to satisfy a narrow requirement.
Do not lose the operational discipline
On-premise deployments drift towards a server under a desk that only one person understands. The practices matter more here, not less, because there is no platform enforcing them.
Containerise, keep infrastructure as code, automate deployment, monitor properly, and make sure at least two people can rebuild it. The absence of a cloud console makes documentation more important rather than less.
The hardware is the cheap part. The person who keeps it running is not.