Explainable AI: Showing Why a Model Made a Decision

A score nobody can explain gets ignored

A credit control team is given a model that flags accounts likely to pay late. Scores come out as numbers between 0 and 1. Within a month the team is back to their own judgement, because nobody could answer the simplest question from a manager: why is this customer flagged?

That is the practical case for explainability, before any regulation enters the picture. People act on predictions they understand. They quietly work around ones they do not. A slightly less accurate model that staff trust will usually produce better business results than a black box they ignore.

Different people need different explanations

Building one explanation for all of them fails every audience. A SHAP chart is useful to a data scientist and meaningless to a customer who has just been declined.

The main explanation methods, in plain terms

• Inherently interpretable models. Linear models, small decision trees and scorecards show their reasoning directly. Often the best choice for regulated decisions.
• Global feature importance. Which inputs matter most across all predictions. Good for sanity checks and management reporting.
• Local attributions, such as SHAP values. For a single prediction, how much each input pushed the score up or down. The workhorse for per-case explanations.
• Counterfactual explanations. The smallest change that would flip the outcome, such as two fewer late payments. Very useful for customers and staff.
• Example-based explanations. Showing similar past cases and what happened to them. Intuitive for experienced staff.

For tabular business data, a gradient-boosted tree model with SHAP explanations covers most needs. Libraries in Python make this straightforward to produce; the real work is translating the output into language people use.

What explanations cannot tell you

It is worth being blunt about the limits, because explainability tools are often oversold.

1. They explain what the model does, not what causes the outcome in the real world. A feature with high importance may be a proxy for something else.
2. Different methods can give different explanations for the same prediction.
3. Correlated inputs share credit unpredictably, so one of two related features can look unimportant when it is not.
4. An explanation can look reasonable while the model is still unfair to a group, so bias testing is a separate job.
5. Explanations for complex models are approximations, however precise the numbers look.

An explanation makes a model easier to question. It does not make the model right.

Regulation and explainability

If a model makes or significantly influences decisions about people, such as credit, hiring, insurance or access to services, explanation stops being optional. UK GDPR gives individuals rights around automated decisions, and the EU AI Act, with obligations phasing in, sets transparency, documentation and human oversight requirements for high-risk uses.

We are not lawyers and this is not legal advice, but the engineering implications are consistent: log each decision with its explanation and model version, document how the model works and was tested, and keep a meaningful human route to challenge an outcome. Our guide to EU AI Act compliance covers the obligations in more depth.

When to choose a simpler model instead

Sometimes the honest answer to an explainability requirement is to give up a little accuracy. If a scorecard with eight factors performs within a whisker of a complex model, the scorecard is easier to explain, easier to defend, easier to monitor and cheaper to maintain.

We run that comparison routinely. For decisions about individuals, we lean towards the interpretable model unless the accuracy gain is large and the value of that gain is clear. Clients sometimes expect the reverse from a machine learning firm. It is still the right call more often than not.

Building explanations into the product

Explanations work best when designed into the screen or letter where the prediction appears, not added as a report later.

• Show the top reasons next to the score, in business language, not feature names
• Hide explanation factors that staff cannot act on, or mark them clearly
• Store the explanation with the prediction so it can be retrieved later exactly as shown
• Let staff record when they disagree, which doubles as useful training data

SpiderHunts treats this as part of the interface design in any machine learning build that touches decisions about customers. If a human then reviews borderline cases, the explanation is what makes that review fast; see human review queues for low-confidence predictions.

Frequently asked questions