Why AI Features Get Worse, and How to Notice

Conventional software does not do this

Ordinary software keeps doing exactly what it did. An AI feature that was 94% accurate in March can be materially worse in September with nothing changed on your side.

That is the single biggest operational difference, and it is why AI maintenance costs more than conventional maintenance.

Three causes of drift

1. Your inputs change. New document formats, new customer types, new product lines, seasonal language.
2. Your ground truth changes. A policy is updated, so answers that were right are now wrong.
3. The model changes. Providers update models, sometimes with subtle behavioural differences.

What we monitor

• Correction rate by category — the earliest reliable signal
• Refusal rate — a jump usually means retrieval has broken rather than the model
• Output length distribution, which shifts before quality visibly does
• Confidence distribution — more items landing in the review band
• Scheduled evaluation runs against the fixed test set

Uptime dashboards give false comfort with AI. A model that is up and answering badly is worse than one that is down, because nothing alerts.

Refresh the evaluation set

A test set built in January describes January. Add new real cases periodically, especially ones the system got wrong, and retire cases that no longer represent your work.

Hold back a portion you never tune against, or you will eventually optimise for the test rather than for reality.

Budget for it

Twenty to thirty per cent of build cost annually, higher than conventional software. That covers evaluation reruns, threshold tuning, prompt adjustments after provider changes, and refreshing the test set.

A business that budgets nothing ends up with a feature nobody trusts, which is worse than not having built it.

Frequently asked questions