Evaluation Pipelines for AI SaaS Products

Why unit tests are not enough

A SaaS team with good engineering habits changes a prompt to fix one customer's complaint. The tests pass, because tests check that the function returns a string, not that the string is right. Two weeks later a different customer notices that dates are now extracted in the wrong format about one time in twenty.

AI features fail statistically. One example is anecdote. You need a few hundred, scored the same way every time, compared against the last release. That is an evaluation pipeline, and for an AI SaaS product it deserves the same status as the test suite. We have written about evaluating AI output quality in general; this post is about wiring it into how a SaaS product ships.

The parts of a working pipeline

1. A versioned evaluation set. Real inputs, consented or anonymised, with expected outputs or grading criteria.
2. A runner. Code that pushes every example through the actual production pipeline, including retrieval and validation.
3. Scorers. Functions that grade each output, from exact match to model-based judgement.
4. A comparison report. Scores against the previous release, broken down by category, with the worst regressions listed.
5. A gate. A rule that fails the build if key scores drop past a threshold.
6. Production sampling. A regular review of live outputs to catch what the fixed set misses.

Tooling matters less than people think. A Python script, a folder of JSON files and a results table in your database will carry a product a long way. Our note on building a Python evaluation harness shows the minimal version.

Choosing what to score

Use deterministic checks wherever the output allows it. Model-graded scoring is useful for open-ended text, but the grader is itself a model with its own errors. Calibrate it: have a person grade fifty examples and confirm the grader agrees with them most of the time before trusting it at scale.

Building the evaluation set

The set should look like your real traffic, including the ugly parts. Scanned invoices at an angle. Emails in three languages. Customers who paste an entire thread into one field. A set made of tidy examples will tell you everything is fine right up to the point it is not.

• Start with 100 to 300 examples, sampled across customers, document types and difficulty
• Tag each by category so reports show where a change helped or hurt
• Add every production failure a customer reports, once fixed, so it never regresses silently
• Keep a small held-out set you look at rarely, to avoid tuning prompts to the test
• Record who labelled each example and when, because labels have errors too

Consent and isolation apply here. Customer data copied into a shared test set needs permission, anonymisation or replacement with realistic synthetic equivalents.

Fitting it into CI without bankrupting yourself

Running three hundred examples through a full pipeline costs real money and several minutes. Running it on every commit is usually wasteful. A tiered approach works well:

• A fast smoke set of twenty to thirty examples on every pull request that touches AI code or prompts
• The full set before merging to main or before each release
• A nightly run against the current production configuration, to catch provider-side model changes
• An extended set when switching models or making large retrieval changes

Cache results for unchanged examples and configurations. Track evaluation spend as its own line in your metering, because it grows quietly as the set grows.

Setting the gate sensibly

A gate that fails on any drop will fail constantly, because model outputs vary between runs. Set thresholds per metric, with tolerance for noise, and treat some categories as critical. A one-point dip in summary tone is a discussion. Any increase in cross-tenant leak tests or invalid JSON is a hard stop.

The purpose of the gate is not to prove the release is perfect. It is to stop you shipping something measurably worse without noticing.

Keep a person in the loop for borderline results. The report should make it quick to read the ten examples that changed most, because reading outputs is still the fastest way to understand what happened.

When the pipeline tells you less than you think

A fixed set drifts away from reality as customers and use cases change. Scores can rise while real satisfaction falls, because the product is getting better at last year's traffic. Production sampling, user feedback rates and support tickets are the counterweight.

At SpiderHunts we treat the evaluation set as a living asset reviewed monthly, not a one-off artefact. When we build AI features through our machine learning and SaaS work, the pipeline arrives with the first feature, because it is the only honest way to answer 'did that change help?'.

Frequently asked questions