Designing for Trust: Confidence and Sources in AI Features

The goal is calibrated trust, not more trust

Teams often ask how to make users trust their AI more. It is the wrong question. If users trust every answer, they will act on the wrong ones, and one bad outcome can end the relationship. If they trust nothing, they check everything and the feature saves no time.

What you want is calibrated trust: users rely on the answers that deserve it and check the ones that do not. That is a design problem as much as a model problem, and most of the tools for solving it are ordinary interface decisions.

Show where the answer came from

Sources are the single most effective trust device in an AI feature. An answer with a link to the paragraph, row or email it was drawn from lets the user verify in seconds.

• Cite inline, next to the claim, not in a list at the bottom
• Make the citation open the exact passage, highlighted, not just the document
• Show a short snippet on hover so most checks need no click at all
• If the answer has no source, say so rather than inventing a reference

Citations only help if they are real. A model asked to produce references without being given retrieved passages will produce plausible ones that do not exist, which is worse than having none. Citations should come from your retrieval step, attached by code, not written by the model. We cover the engineering side in reducing AI hallucination in integrations.

Turn confidence into a few meaningful states

A raw score like 0.83 means nothing to a user. Worse, model self-reported confidence is often poorly calibrated. What works is a small number of states that each imply an action.

The thresholds behind these states should come from evaluation on your own data, not from a gut feel. Pick them so that items marked verified are right almost all the time, and accept that more items will land in check this as a result. Users forgive being asked to check. They do not forgive being told something was verified when it was not.

Deterministic checks beat model confidence

Wherever you can check an answer with code, do. An extracted invoice total can be compared with the sum of line items. A date can be checked against a plausible range. A customer ID can be looked up. These checks are cheap, reliable and explainable.

1. List the fields or claims that matter most
2. For each, write any rule that can confirm or contradict it
3. Use rule results to decide the displayed state before using model confidence
4. Log which rule failed so users see a reason, not just a warning colour

A message like 'total does not match line items' is far more useful than an amber dot. It tells the user exactly what to look at.

Consider an illustrative case: a property management tool that extracts break clauses and rent review dates from leases. The model finds a rent review date with high self-reported confidence, but the date falls before the lease start. A simple rule catches it, the field shows as check this with the reason, and a surveyor fixes it in ten seconds. Without the rule, the wrong date sits quietly in a report until a review is missed.

Honest language in the interface

Wording matters. 'Based on 3 documents' is better than 'Here is the answer'. 'I could not find this in your files' is better than a guess. Avoid anthropomorphic reassurance such as 'I am confident that', which sounds authoritative and carries no information.

An AI feature that says 'I don't know' at the right moments earns the right to be believed the rest of the time.

Transparency is also becoming a compliance matter. The EU AI Act's transparency obligations, which are phasing in, expect people to know when they are dealing with AI-generated content. Clear labelling and visible sources help with that as a side effect of good design.

Patterns that backfire

• Percentages on everything, which users learn to ignore within a week
• Warnings on every answer, so the real warnings disappear into the noise
• Citations to whole documents of forty pages, which nobody opens
• Hiding uncertainty in a tooltip that only appears on hover
• Letting the model decide its own confidence label with no external check

How we approach it

At SpiderHunts we set trust thresholds from a labelled sample of a few hundred real cases before a feature launches, and we review how often users override each state once it is live. If people routinely edit verified answers, the threshold is wrong. If they never touch check this items, the warning is too cautious. Both are fixable, but only if you are measuring them.

This work usually sits inside our AI integration projects, alongside the evaluation process we describe in how we evaluate whether an AI feature is good enough.

Frequently asked questions