Which Model Should Your Application Use?

The question assumes a single answer

“Which model is best” presumes one workload. Most business applications involve several tasks — classification, extraction, retrieval, generation — and those want different models.

The productive question is: for each step, what is the cheapest model that meets the quality bar for that step?

Routing by difficulty

Test on your data, not on benchmarks

Public benchmarks measure general capability and correlate loosely with performance on your specific job. A few hundred of your own cases with agreed answers settles the question in an afternoon.

That set is also what lets you switch models later without re-testing everything by hand, which matters because better and cheaper options appear every few months.

Design for replacement

• Prompts in configuration, not scattered through code
• The provider abstracted behind a thin interface
• No dependence on provider-specific quirks unless the benefit is large
• The evaluation set rerun automatically on any model change

Latency is a product decision

A user waiting has a different tolerance from a background job. Interactive features need fast first tokens and benefit from streaming; batch processing can use slower, cheaper, stronger models.

Decide the latency budget per feature before choosing, because it eliminates options quickly and prevents a late redesign.

Frequently asked questions