How We Choose the Right AI Model
Last updated:
Why this article names no models
Any article that recommends a specific model by name is wrong within months. Providers ship new versions constantly, prices fall, and a model that was clearly best in one quarter is mid-table by the next.
So we publish the criteria rather than the answer. The criteria have been stable for years; the answer changes at every project kick-off, and we check it again before launch.
1. Capability, measured on your task
Public benchmarks are close to useless for choosing. They measure general ability on academic problems; you need performance on your documents, with your edge cases, judged by your standard.
So we build a small evaluation set first: 50–200 real examples with the correct output written by someone who knows the domain. Every candidate model is scored against it. This takes about two days and is the single highest-value thing you can do on an AI project.
Two days building an evaluation set has saved every AI project we have run more time than it cost, usually by eliminating a plausible option early.
2. Cost per completed task
Not cost per token — cost per unit of work that came out right. A cheaper model that needs two attempts and a longer prompt can be more expensive than a capable one that succeeds first time.
- Tokens in, including any retrieved context, which is usually the bulk
- Tokens out
- Retries caused by failures and refusals
- Human correction time, priced at a real hourly rate
That last line dominates whenever accuracy differs. A model five percentage points more accurate can be worth several times the per-token price once someone has to fix the errors.
3. Latency, against a real budget
Decide the budget before choosing: a user-facing suggestion needs to feel instant, a queued document extraction can take a minute, an overnight batch can take an hour.
Latency also varies by time of day and by provider load. We measure at the times your system is actually busy rather than at 11pm when everything is fast, and we test the slow path deliberately.
4. Where the data goes
- Which jurisdiction processes the request, and is that acceptable to you?
- Is your input used for training, and can that be contractually excluded?
- How long is data retained, and can retention be turned off?
- Does the provider offer the certifications your customers ask about?
- Is a self-hosted option available if the answer to any of the above is no?
For regulated clients this axis outranks capability. A better model you cannot use is not a better model, and we would rather establish that in week one than in a security review three months later.
5. Exit cost
How hard is it to move? Anything provider-specific — a proprietary tuning format, a bespoke tool-calling shape, a hosted vector store — is a cost you pay later, at the least convenient moment.
We keep prompts, evaluation sets and retrieval data in our own storage, and the provider behind one interface. Then a model change is a configuration change plus a re-run of the evaluation set, which is a day rather than a rewrite.
How the decision usually lands
| Task shape | What tends to matter most |
|---|---|
| High-volume classification | Cost per task, then latency |
| Document extraction | Accuracy on your formats, then cost |
| Customer-facing drafting | Quality and tone, then latency |
| Agentic multi-step work | Instruction-following and tool use |
| Regulated or sensitive data | Data handling, before anything else |
We usually run two candidates through the evaluation set and pick on the numbers, then re-check at launch and quarterly afterwards.
Frequently asked questions
Do you have a preferred provider?
Can we use more than one?
How often should we re-evaluate?
What about open-weight models we host ourselves?
Wondering whether AI would actually help here?
Describe the task and we will tell you plainly whether it is a good fit, including when the honest answer is that ordinary software would serve you better.