Choosing Between a Cloud ML Platform and a Custom Stack

The platform demo is always impressive

Every cloud provider's machine learning platform demo follows the same arc: notebooks, one-click training, a model registry, an endpoint, a monitoring dashboard, all in one console. It looks like the answer to every MLOps problem at once.

For a large data science team, it often is. For a 90-person business with one data analyst and a churn model, it can be a lot of expensive machinery around a job that a scheduled Python script would do. The decision deserves more thought than which console looked nicest.

What each option actually means

Neither column is the right answer in general. The right answer depends on a few facts about your organisation, and those are easy to establish.

When a managed platform is the better choice

• Several data scientists or teams building and deploying models independently
• Ten or more models in production, or a clear plan to get there
• Heavy training workloads needing GPUs on demand
• A company already committed to one cloud, with engineers who know it well
• Regulated environments where the platform's built-in audit and access controls save real work
• No appetite to maintain infrastructure, and budget to pay someone else to

In those situations the shared tooling, permissions and standardisation are worth the fees and the lock-in. Rebuilding them yourself would cost more.

When a custom stack is the better choice

• One to five models, mostly tabular data, mostly batch predictions
• A small team, often one or two people who also do other work
• Low or irregular prediction traffic, where always-on endpoints waste money
• An existing, well-run application stack the models can slot into
• A desire to keep the option of moving providers

A typical small custom stack is not exotic: training code in a Git repository, MLflow or a simple table for tracking models, a scheduled job for batch scoring, predictions written to your main database, and monitoring through your existing tools. Our post on choosing an MLOps stack covers the components in more detail.

The costs people miss on each side

Platforms hide some costs in their convenience. Endpoints left running, notebook instances nobody shut down, storage of every experiment and data egress all turn up on the bill. The learning curve is real too; platform concepts do not transfer between providers.

Custom stacks hide costs in people. Someone has to patch servers, upgrade libraries, keep the scheduler running and document it all. If that person leaves, the stack can become a mystery quickly. We have inherited a few custom set-ups that were elegant and completely undocumented, which is its own kind of lock-in.

Platforms lock you into a vendor. Undocumented custom stacks lock you into a person. Pick the dependency you can manage.

A decision checklist

1. Count your production models today and realistically in two years.
2. List who will maintain them and what they already know.
3. Note your prediction pattern: batch, real-time, or both, and at what volume.
4. Check which cloud you are already on and how deep that commitment is.
5. Estimate a year of platform fees at your expected usage, including endpoints.
6. Estimate the engineering days to build and maintain a lean custom equivalent.
7. Weigh any compliance or audit requirement the platform would satisfy for you.

If you are comparing providers specifically, Azure ML versus AWS machine learning sets out the differences between two of them.

Questions to ask before signing up to a platform

Platform sales conversations tend to focus on features. The questions that decide whether you will be happy a year later are more mundane.

• What does an idle endpoint cost per month, and can it scale to zero?
• Which parts of our pipeline would be written in platform-specific formats?
• Can we export trained models in a standard format and run them elsewhere?
• How are permissions, secrets and audit logs handled, and does that match how we already work?
• What happens to our bill if prediction volume grows tenfold?

If the answers are vague, run a small pilot with one model and watch the invoice for two months before committing the rest.

A hybrid is often sensible

It is not all or nothing. Many businesses use managed services for the parts that are painful to run, such as GPU training jobs or a data warehouse, while keeping model code, tracking and batch scoring portable and simple. Writing training code as plain Python rather than platform-specific pipelines keeps the exit door open.

SpiderHunts has built both. For most SME machine learning projects we start lean and portable on the client's existing cloud, and recommend a platform when team size or model count makes shared tooling worth the fees. We would rather tell you that up front than migrate you later.

Frequently asked questions