TL;DR
- 7 stages: problem definition → data collection → data preparation → feature engineering → model training → evaluation → deployment + monitoring
- Data preparation is typically 40–60% of the total project time: this surprises most clients
- Never use accuracy alone as a success metric; use precision, recall, and business metrics
- Deployment is not the finish line: monitoring and retraining are ongoing requirements
- Python + scikit-learn for most structured data problems; PyTorch/TensorFlow for deep learning
Building a custom machine learning model is a structured engineering process. It is more demanding than using a pre-built AI tool. But it produces results that are specifically optimised for your data, your problem, and your accuracy requirements. (If you would rather hand the build to a team, see our custom machine learning development service.)
This guide walks through every stage. That includes the parts that are rarely discussed in tutorial content but consume most of the real project time.
Stage 1: Define the Problem Precisely
The most common reason ML projects fail is a poorly-defined problem. Before touching data, you need to answer four questions:
- What decision am I trying to make or improve? (e.g., "Which customers are likely to churn in the next 30 days?")
- What does a correct output look like? (A probability score? A category? A number?)
- What is the cost of a wrong prediction? (False positives vs. false negatives: these have different costs in different contexts)
- What data is available and when? (You can only use features that are available at prediction time)
A well-defined problem statement: "Predict whether a customer will cancel their subscription in the next 30 days based on their last 90 days of usage data and support interactions, so we can trigger a retention campaign. We care more about catching churners (recall) than about avoiding false alarms (precision), as the campaign cost is low."
Stage 2: Collect and Assess Your Data
Identify all data sources relevant to your problem. For each, assess:
- Volume: Do you have enough examples? (Minimum: 500–1,000 labelled examples; more is better)
- Quality: What is the missing data rate? Are there labelling errors?
- Recency: Is the data from the relevant time period? Old data may reflect a different world.
- Balance: For classification, are both classes represented? A 99%/1% split creates a challenging imbalanced learning problem.
This stage often reveals unexpected data quality issues that must be resolved before training. Budget time for it.
Stage 3: Data Preparation
This stage is where most of the real work happens. Typical tasks:
- Handling missing values: imputation (fill with mean/median/mode), removal, or flagging
- Encoding categorical variables: converting text categories to numbers (one-hot encoding, label encoding)
- Scaling numerical features: normalising or standardising so large-valued features do not dominate
- Handling outliers: deciding whether extreme values should be capped, removed, or treated specially
- Train/validation/test split: typically 70% training, 15% validation, 15% test: with the test set held back until final evaluation
All transformations applied to training data must be applied identically to new data at inference time. This is a common source of production bugs if not designed carefully.
Stage 4: Feature Engineering
Features are the inputs your model will use to make predictions. Raw data is rarely in the optimal form. Feature engineering creates new, more informative representations:
- Converting raw timestamps to "days since last purchase" or "day of week"
- Aggregating transaction history into "average spend per month" and "number of transactions in last 30 days"
- Creating interaction features: "days since purchase × number of support tickets"
- Ratios and rates: conversion rate, average order value, return rate
Stage 5: Model Training and Selection
Train multiple candidate algorithms and compare them on your validation set. Common choices for structured/tabular business data:
| Algorithm | Strengths | When to use |
|---|---|---|
| Logistic Regression | Fast, interpretable, good baseline | Binary classification, first model |
| Random Forest | Handles missing data, feature importance | Most tabular classification/regression |
| XGBoost / LightGBM | State-of-art on tabular data, fast | Competition-grade accuracy on structured data |
| Neural Networks | Best for images, text, audio | Unstructured data; complex pattern recognition |
| LSTM / Transformer | Sequential/time-series data | Demand forecasting, anomaly detection in sequences |
Stage 6: Evaluation
Never evaluate on your training data. Always use the held-back test set. Choose metrics that reflect the actual business cost of errors:
- Accuracy: Good when classes are balanced; misleading when they are not
- Precision: Of the positive predictions, how many were correct? (Important when false alarms are costly)
- Recall: Of all actual positives, how many did you catch? (Important when missing a positive is costly)
- AUC-ROC: Overall discrimination ability across all decision thresholds
- Business metrics: Revenue impact, cost savings, customer satisfaction: the numbers that actually matter to the business
Stage 7: Deployment and Monitoring
Deployment options depend on how the model will be used:
- REST API: Most common. Model is served via FastAPI or Flask; applications call the endpoint to get predictions. Scales well.
- Batch processing: Model runs on a dataset overnight (e.g., generating churn scores for all customers every morning)
- Embedded: Model baked into a mobile or desktop application (requires model compression)
Post-deployment monitoring is non-negotiable. Track: prediction accuracy (if you have ground truth feedback), input data distribution (data drift detection), and inference latency. Set up automated alerts for significant accuracy drops. Plan for quarterly retraining at minimum.
Typical Project Timeline and Budget
| Complexity | Example problem | Timeline | Budget range |
|---|---|---|---|
| Simple classification | Lead scoring, spam filter | 4–8 weeks | £8k–£18k |
| Demand forecasting | Inventory / revenue prediction | 8–16 weeks | £15k–£35k |
| Recommendation engine | Product recommendations, content | 10–20 weeks | £20k–£50k |
| Computer vision / NLP | Image classification, document extraction | 12–24 weeks | £25k–£80k+ |
Ready to Build Your Custom ML Model?
We work with you from problem definition through to production deployment. We also stay engaged for monitoring and retraining.
Start Your ML Project See ML Services