How We Measure Whether It Worked

The measurement everyone forgets to take

The most valuable measurement on a software project happens before it starts. How long does the process take now? How many errors a week? How many of those reach a customer?

Take it in week one or you will never have it. After launch, everyone's memory of the old process is unreliable in a predictable direction: people underestimate how bad it was, which makes a genuinely successful project feel unremarkable.

“It feels faster” is not a result you can take to a board. “Order processing went from 13.7 hours a week to 2.1” is.

Three tiers, in the order they arrive

1. Adoption — weeks 1–4. Is anyone using it? Daily active users against the number who should be, and how much of the old process is still running.
2. Efficiency — months 1–3. Time per instance, error rate, cycle time, manual interventions per hundred records.
3. Business outcome — months 3–12. Orders handled per head, quotes converted, cost per transaction, whatever the original case promised.

Judging on tier three at week six is the classic mistake and produces false disappointment. Judging on tier one at month nine is the opposite mistake and misses a system that is being used but is not helping.

Adoption is the early warning system

Low adoption in the first fortnight is the single most reliable predictor of a failed project, and it is almost never about the software's quality. It is training, a missing edge case, or one influential person who prefers the old way.

• What percentage of expected users logged in this week?
• What percentage of transactions went through the system rather than around it?
• Is the old spreadsheet still being edited? (Check the modified date — it is the most honest metric available.)
• Which single feature accounts for most of the usage?

That last one is interesting almost every time. There is usually one feature nobody prioritised that turns out to be why people open the thing.

Instrumenting without building an analytics project

You do not need a data warehouse. Three things cover most of it: server-side event logging for the handful of actions that matter, a weekly summary query, and a simple dashboard anyone can open.

We build that into every project by default rather than treating measurement as a separate phase. It costs a few days and it is the difference between knowing and believing.

Reporting back honestly, including the misses

At 30, 90 and 180 days we send a short written report comparing the baseline to now — including where the target was missed and what we think the reason is.

One recent report showed processing time down 78% against a 60% target, and error rate essentially unchanged against a target of halving it. The cause was a data quality issue upstream that our system faithfully passed through. Saying so produced the next project; hiding it would have produced a quiet non-renewal.

When the numbers say it did not work

It happens. The usual causes, in order: the process changed underneath the software, adoption never reached critical mass, or the original business case measured the wrong thing.

All three are recoverable if they are named early. What is not recoverable is a year of nobody looking, followed by a decision to write the whole thing off because it “never really worked” — a conclusion nobody can check, because nobody took the baseline.

Frequently asked questions