Automation, AI and Machine Learning in One System

Three different tools that get called 'AI'

Automation, machine learning and generative AI are regularly sold as one thing. They are not. They fail in different ways, cost different amounts and need different kinds of care. The systems that work well in a business use each of them for what it is good at, and keep them in their lanes.

We have written separately about combining automation and AI in one project, where the point was that most of a real project is ordinary automation. This post adds the third layer, trained machine learning, and describes how the three fit together in one system.

What each layer is for

Our rule of thumb when designing a step: use the least clever tool that can do it reliably. If a rule will do, write a rule. If a rule cannot capture the pattern but your history can, train a model. Reach for a language model when the input is language.

A worked example: order handling at a wholesaler

Picture a 50-person wholesaler receiving around 250 orders a day, many as emailed PDFs or free-text emails, from trade customers with credit accounts. Here is how the three layers share the work.

1. Language model: reads each incoming email or PDF, extracts customer, product lines, quantities and requested delivery date, and gives a confidence score per field
2. Rules: check product codes exist, prices match the customer's agreed price list, and stock is available; anything that fails goes to review
3. ML model: scores the order for payment risk using the customer's order and payment history, and flags unusual quantities compared with that customer's pattern
4. Rules: route by thresholds, so clean, low-risk orders post straight to the ERP while low-confidence, high-risk or unusual ones go to a person
5. Person: sees the order, the extracted fields beside the original document and the reasons for the flag, then approves or corrects in one screen
6. Language model: drafts the confirmation or query email back to the customer for a quick check

Notice what the language model never does: it never decides the price, never judges credit and never posts to the ledger. Those are jobs for rules and trained models, which are predictable and auditable.

The plumbing all three share

Combining the layers only works if they sit on common foundations. Without them you have three separate projects that happen to touch the same data.

• One event log. Every step records what came in, what each layer decided and why, so any order can be traced end to end
• One review queue. People handle exceptions from all three layers in the same place, with the same controls
• Monitoring per layer. Extraction confidence, model drift and rule failure rates are tracked separately, because they degrade for different reasons
• Cost tracking per task. Language model calls are the variable cost, so they are counted per order
• Corrections captured. Every human fix becomes evaluation data for the language model and training data for the ML model

The review queue is the most important screen in the system. It is where the three layers admit what they do not know.

The order we build it in

We almost never build all three layers at once. The usual sequence starts with the plumbing and rules, because they deliver value immediately and give the other layers somewhere to live. Language model extraction comes next, since it usually removes the most manual work. The trained model comes last, once the system has been capturing clean outcome data for long enough to train on.

This ordering also protects the budget. If the business case is already met after the first two stages, the ML layer can wait. Our automation and machine learning teams work on the same codebase for exactly this reason.

How we decide which layer a step belongs to

For each step in a process we ask a small set of questions, in order.

1. Can the correct answer be written as a rule someone would sign off? Use a rule.
2. Is the input unstructured text, speech or a document? Use a language model to turn it into structured data, then return to rules.
3. Is it a prediction that experienced staff make from patterns in past cases? Consider a trained model, if the history exists.
4. Would a wrong answer cause harm before anyone notices? Add a human review step, whichever layer made the call.

SpiderHunts works through this with the people who do the job, because they know which decisions are genuinely rules and which only look like rules until the exceptions appear.

When one combined system is too much

Not every business needs all three layers joined up. If orders arrive in a structured feed, there is nothing for a language model to read. If volumes are low, a trained model has too little data and an experienced person judges risk better. And if two processes share no data or people, building them as one system just couples things that should fail independently. A combined system earns its complexity when the same case genuinely passes through reading, rules and prediction on its way to a decision.

Frequently asked questions