Product Analytics for AI Features
Last updated:
Standard analytics miss the point of AI features
Most SaaS teams already track feature usage: clicks, views, sessions, funnels. Our general guide to product analytics and what to track still applies. But for an AI feature, knowing that someone clicked generate tells you almost nothing. The output could have been perfect, useless or wrong.
The questions that matter are different. Did the user use what the AI produced? How much did they have to change it? Did they come back and use it again? What did it cost to produce the outputs they kept? Answering those needs events that most analytics set-ups do not have.
The core events to instrument
| Event | Captures | Why it matters |
|---|---|---|
| ai_output_shown | Feature, input size, model route, latency, cost | Baseline for every ratio |
| ai_output_accepted | Accepted as is | The clearest signal of value |
| ai_output_edited | Edit size relative to output length | Measures how much work the AI saved |
| ai_output_rejected | Dismissed or deleted, with optional reason | Catches failures users do not report |
| ai_output_regenerated | Retry requested | First result was not good enough |
| ai_feedback_given | Thumbs up or down, comment | Useful but sparse and biased |
| ai_handover | Passed to a person or manual path | Shows where the AI could not cope |
Give every output a unique ID and attach it to all of these events. That lets you join what the user did to what the system produced, the prompt version, the retrieved sources and the cost. Without the ID, you have counts but no way to investigate them.
The metrics that come out of those events
- Acceptance rate: accepted plus lightly edited outputs, divided by outputs shown
- Average edit size: how much of each output users change, tracked over time
- Retry rate: regenerations per output shown, which rises when quality slips
- Time to accept: very fast accepts on long outputs may mean rubber-stamping
- Repeat use: share of users who use the feature again within seven days
- Cost per accepted output: total AI cost divided by outputs actually used
The last one is the metric finance teams care about and product teams often do not calculate. A feature where only a third of outputs are accepted costs three times what the per-call cost suggests. That changes decisions about pre-running features in the background, model choice and pricing.
Explicit feedback is the weakest signal
Thumbs up and down buttons are easy to add and tempting to rely on. They are used by a small share of users, mostly when something is very good or very bad, and they tell you little about the middle. Treat them as a source of examples to read, not as a quality metric.
What users do with an output is far more honest than what they say about it.
Behavioural signals such as acceptance, edits and retries are captured for every output, without asking anything of the user. Build your dashboards on those.
Sampled quality scoring
Behaviour tells you whether users kept an output, not whether it was correct. A user can accept a subtly wrong summary. So combine behavioural analytics with a sampled quality score.
- Take a random sample of outputs each week, stratified by feature and customer segment
- Score them against a short rubric, by a person or an evaluated model grader checked against people
- Track the score over time next to acceptance rate
- Investigate whenever the two diverge, since high acceptance with falling quality means rubber-stamping
This is where analytics and evaluation meet. The same rubric should be used in pre-release testing, so a change that scores well offline can be confirmed in production.
Segments that reveal the real story
Averages hide everything interesting in AI features. Always split by:
- Customer or account, because data quality varies enormously
- Input type, such as document format, language or length
- User role, since the same feature helps some roles far more than others
- Prompt or model version, so you can see the effect of each change
- Tenure, because new users and experienced users judge outputs differently
In an illustrative case, a contract review feature might show a respectable overall acceptance rate that turns out to be very high on standard agreements and poor on anything scanned. That single split tells you where to invest, and the overall figure never would.
Vanity metrics to stop reporting
Total generations, AI button clicks and tokens processed look impressive in a board deck and mislead everyone in it. Generations can rise because quality fell and users are retrying. Clicks can rise because the button moved. Report accepted outputs and cost per accepted output instead.
When SpiderHunts instruments an AI feature, we set up the output ID, the core events and a weekly quality sample in the first release. It is a modest amount of work inside a data science or product engagement, and it turns every later argument about whether the feature is good into a question with an answer. For operational health such as latency and error rates, pair this with monitoring and alerting for AI apps.
Frequently asked questions
What metrics should you track for an AI feature?
Are thumbs up and down ratings useful for AI features?
How do you measure how much time an AI feature saves?
Which analytics tools work for AI feature tracking?
What is a good acceptance rate for an AI feature?
Cannot tell whether your AI feature is working?
Share your current event tracking and a description of the feature. We will list the events you are missing and what they would tell you.
Related services
What we build for problems like this one