Decide the measures before you build
Measuring afterwards is always disputed. Deciding what counts as success while everyone is still optimistic is what turns a later argument into arithmetic.
We ask clients to capture the baseline for two weeks before anything changes, whoever builds the system.
The five
- Minutes per unit of work — per order, per invoice, per enquiry
- Errors per hundred units, however roughly you can capture it
- Elapsed time from start to finish, which is different from effort and usually the bigger win
- Review rate — the proportion still needing a human, which should fall
- Support contacts about that process, which reveals whether customers noticed
Review rate is the one people forget
A review rate that is flat after three months means either the thresholds were never tuned or the corrections are not feeding back. Both are fixable and neither fixes itself.
We report it monthly during the first quarter so the trend is visible while it can still be influenced.
What to do when a number moves the wrong way
- Review rate rising — usually an upstream format change; check field completeness
- Errors rising — a new case class the rules do not cover; add it to the exception design
- Elapsed time unchanged — the bottleneck moved rather than disappeared; look at the next step
- Support contacts rising — customers are experiencing something you cannot see internally
Be willing to conclude it did not work
Set a review at three and six months with the same measures, and decide in advance what result would mean widening, iterating or stopping.
Stopping something that did not work is a good outcome. The failure is a system that persists for two years because nobody wanted to say it was not helping.