Two different questions
Explainability usually gets discussed as understanding a model overall - which features matter most, how it behaves in general. That is a question for the people who own it.
The person declined, flagged or charged more has a different question: why me, and what can I do about it? Global feature importance does not answer that, and reading it out would be worse than saying nothing.
What an individual explanation should contain
- The main factors in this specific case, in plain language, ordered by influence.
- What would have to change for a different outcome, where that is actionable.
- Whether a person was involved, and how to ask for one.
- How to correct the underlying data if it is wrong.
- Who to contact, and what happens next.
That fourth point matters more than it gets credit for. A meaningful share of adverse automated decisions trace to incorrect data rather than a wrong model, and a correction route resolves those without anyone appealing.
Language that helps rather than deflects
| Unhelpful | Better |
|---|---|
| Your application scored below our threshold | The main factors were a short account history and two recent missed payments |
| Our system made this decision | This was decided automatically. You can ask for a human review |
| We cannot disclose our model | We can tell you the main reasons and how to have them reviewed |
| Improve your creditworthiness | A longer payment history with us would change this - we can review again in six months |
The right-hand column is not more revealing about the model. It is simply more useful to the person reading it.
Without giving away the model
There is a genuine tension where explanation enables gaming - fraud and abuse detection in particular. It is narrower than it is usually claimed to be.
Practical approach: give reasons at the level of broad factors rather than exact thresholds, avoid stating how much a change would move the score, and be more conservative for decisions where gaming carries real cost. For most business decisions - a declined credit limit, a flagged claim - the risk is modest and the obligation is real.
Where you genuinely cannot explain without enabling abuse, that is an argument for a human in the loop, not for an unexplained automated decision.
Design it before you need it
Explanation capability has to be built in. Recovering why a decision was made six months ago requires the inputs, the model version and the score to have been stored at the time.
Retro-fitting that is difficult and sometimes impossible. Storing the explanation alongside the decision, at the moment it is made, is far cheaper than reconstructing it later under pressure. Our note on explainable AI covers the technical side.
If you cannot say why, you are not ready to decide automatically.