Combining Machine Learning Predictions With LLM Explanations

A score on its own rarely changes behaviour

A churn model flags an account at 0.82. The account manager sees a red badge in the CRM and has no idea why, what to say, or whether this is the same account that was red last month and renewed anyway. The badge gets ignored within a fortnight.

This is the quiet failure of many predictive projects. The model is accurate, the dashboard works, and nothing changes because the people who need to act cannot see the reasoning. Explanations were always possible with feature importance charts, but a waterfall plot of SHAP values is not something a sales team reads between calls.

Language models are very good at the missing step: turning a structured set of facts into two sentences a person can act on. The key is keeping each tool in its lane.

Who does what

The LLM is the narrator, not the detective. It explains what the model found; it does not decide what happened.

How the pipeline works

1. The ML model scores the record and returns a calibrated probability
2. An explainer computes the top contributing features for that specific prediction
3. Code translates feature names into business terms, such as 'logins_30d_delta' to 'logins down 60% over 30 days'
4. Code adds relevant context from source systems
5. The LLM receives only these facts, plus a list of approved actions, and writes a short explanation in structured output
6. A validation step checks that every reason mentioned is in the supplied driver list

Step three is where much of the value sits and where we spend real time with clients. Raw feature names are meaningless to users, and badly translated ones produce explanations that sound confident and say nothing.

What a good explanation looks like

For the churn example, a useful output reads something like: 'Higher risk than usual, mainly because weekly logins have fallen by more than half since the admin user left in July, and two support tickets about invoicing are still open. Suggested: call the new admin and resolve the invoicing tickets before the renewal date on 30 October.'

• Names two or three drivers, not ten
• Uses the business's own language and real values
• Connects to a concrete, approved action
• Avoids claiming certainty the probability does not support
• Stays short enough to read in a CRM sidebar

The same pattern suits credit risk notes, demand forecast commentary, predicted maintenance alerts and fraud review queues. Our churn prediction model post covers the modelling side of that example.

The risk: explanations that sound right and are not

An LLM asked 'why might this customer churn' without constraints will produce a plausible story from general knowledge. That story may have nothing to do with the model's actual reasoning, and because it reads well, people believe it.

• Never ask the LLM to guess reasons. Give it the drivers and forbid others.
• Validate mentions. Check in code that each reason cited maps to a supplied driver.
• Remember SHAP explains the model, not the world. A driver is correlated with the prediction; it is not proof of cause. Word explanations accordingly.
• Watch for proxies. If postcode drives a lending score, the explanation surfaces a fairness problem rather than an insight.
• Keep the numbers visible for users who want to check.

In regulated decisions, such as credit or employment, explanations may carry legal weight, and the EU AI Act's transparency and oversight duties for high-risk systems apply. There, templated explanations with fixed wording are sometimes a safer choice than generated ones.

When you do not need the LLM at all

If there are only a handful of drivers and actions, templates do the job: 'Risk raised by: falling usage, open tickets.' They are deterministic, free, easy to audit and never invent anything. The LLM earns its place when drivers combine in many ways, when context from several systems matters, or when explanations need to read naturally in different languages.

It is also worth asking whether a simpler, more interpretable model would remove the need for explanation work entirely. A well-built scorecard is sometimes the better product.

Cost is rarely the deciding factor. Explanations are short, generated only for records someone will look at, and can be produced in a nightly batch alongside the scores. A sales team reviewing 200 flagged accounts a week is a trivial model bill. The real cost is the upfront work of mapping features to language and agreeing the approved actions with the people who will use them.

How SpiderHunts builds this

When SpiderHunts delivers a predictive model, we now routinely ask who acts on it and what they need to see. Where explanations help, we build the explainer, the feature-to-language mapping and the constrained LLM step together, and test explanations with the users before launch, measuring whether they take the suggested action more often.

It sits across our machine learning and data science work. If you are weighing where to use which kind of model, see classic ML vs LLM for classification.

Frequently asked questions