Building Evaluation Datasets From Real Business Cases

Public benchmarks do not know your business

A model can top every public leaderboard and still misroute your support tickets, because the leaderboard never saw your product names, your refund rules or the way your customers write when they are annoyed.

The only reliable way to know whether an AI feature works for you is to test it on your own cases, with answers your own people agree are correct. That collection is an evaluation dataset. It is the most durable asset in an AI project: models change every few months, and a good dataset lets you test each new one in an afternoon.

The good news is that most businesses already hold the raw material. Past tickets with the category an agent chose. Invoices with the figures someone keyed in. Emails with the reply that was actually sent.

Where the cases come from

Be sceptical of historical labels. The queue a ticket ended in is better than the one it started in, and a value in the ERP may have been corrected later. Spot-check a sample before trusting any source as ground truth.

Sampling: random is not enough

A purely random sample of 200 tickets from a business where 70% are 'where is my order' produces a dataset that is 70% easy. The model scores brilliantly and fails on the cases that actually cost money.

• Stratify by category so each important type has at least 15 to 20 examples
• Deliberately include hard cases: ambiguous requests, multiple issues in one message, poor scans, other languages
• Include high-stakes cases such as complaints, legal threats and large orders, even if rare
• Include cases where the right answer is 'I don't know' or 'escalate', which models are naturally bad at
• Span time periods, so seasonal patterns and old product names appear

Keep a note of the real-world frequency of each category too, so you can weight results back to reflect what production traffic will look like.

Labelling that people agree on

The expensive part of an evaluation set is not collecting cases. It is agreeing what correct means.

1. Write labelling guidelines with definitions and borderline examples for each category or field
2. Have two people label the same 30 to 50 cases independently
3. Compare, discuss disagreements and update the guidelines
4. Only then label the rest, with a second check on a sample
5. Record ambiguous cases explicitly rather than forcing a label

If two experienced staff agree only 80% of the time, no model will reliably reach 95% against either of them, and the real problem is an unclear process. Discovering that is a useful outcome in itself. For free-text outputs, label with a short rubric of must-include facts and must-avoid mistakes rather than a single perfect answer.

Personal data and privacy in evaluation sets

Real cases contain real customers. Before cases go into a shared dataset or a vendor's evaluation tool, redact names, contact details and account numbers, or replace them with consistent placeholders so the text still reads naturally. Keep the dataset under the same access controls as the source system, and record its purpose so its retention can be justified under GDPR.

Our post on handling personal data sent to LLMs covers the redaction techniques in more detail.

Keeping the dataset honest over time

• Hold back about a fifth of cases that nobody looks at while tuning prompts, and score on them only before release
• Add every production failure as a new case, with the corrected answer
• Re-check old labels when business rules change, such as a new refund policy
• Version the dataset, so a score is always reported against a named version
• Retire cases about discontinued products rather than letting them skew results

The held-out portion matters more than it sounds. Tune a prompt against the same 200 cases for a month and it will fit those cases, not the business.

Store the dataset somewhere boring and shared: a versioned folder in the repository or a table in your database, not a spreadsheet on one person's laptop. Each case needs an ID, the input, the expected output or rubric, the category, the source, and a note on why it was added. That last field sounds fussy until someone asks, a year later, why a strange-looking case is in the set and whether it can be deleted.

How SpiderHunts builds these with clients

On a typical SpiderHunts project, we spend the first week pulling a few hundred real cases from the client's systems, drafting labelling guidelines, and running a short labelling session with two people who do the job every day. It is often the most revealing week of the project, because it surfaces rules nobody had written down.

The dataset then drives every decision that follows: which model, which prompt, what confidence threshold, and whether the feature is ready. We describe how scores turn into go or no-go calls in how we evaluate whether an AI feature is good enough, and it is part of our machine learning and AI integration work.

Frequently asked questions