The question to ask first
When an AI feature gives a disappointing answer, there are only two underlying causes. Either it did not know something it needed to know, or it knew and expressed it wrongly.
Retrieval fixes the first. Fine-tuning fixes the second. Teams routinely reach for fine-tuning to solve knowledge problems, spend a month on it, and end up with a model that is confidently wrong in a more polished way.
What retrieval actually does
Retrieval-augmented generation looks up relevant material at question time and puts it in the prompt. Your documents stay where they are, updates are immediate, and the answer can cite its source.
- New or changed documents are available instantly — no retraining
- Answers can be traced back to a source, which matters for trust and audit
- Access control can be enforced at retrieval time, per user
- Cost is per query rather than a large one-off
The trade-off is that every query carries retrieval cost and latency, and quality depends heavily on how documents are chunked and indexed. That work is where the skill sits.
What fine-tuning actually does
Fine-tuning adjusts the model's weights on examples of the behaviour you want. It is very good at teaching a consistent output format, a house tone, or a classification scheme with your specific labels.
It is poor at teaching facts. The knowledge gets blurred into the weights, it cannot be cited, it cannot be updated without retraining, and the model will happily generalise beyond what you taught it.
A straight comparison
| Retrieval | Fine-tuning | |
|---|---|---|
| Fixes | Missing knowledge | Wrong behaviour or format |
| Setup effort | Days to weeks | Weeks, plus dataset preparation |
| Updating | Add the document | Retrain |
| Citations | Yes | No |
| Typical cost | £8,000–£30,000 build, then per query | £15,000–£60,000 including data work |
| Access control | Straightforward per user | Not possible — it is in the weights |
When fine-tuning genuinely wins
Three cases in our experience. First, a rigid output format that prompting keeps drifting away from at volume. Second, a specialist classification scheme with thousands of labelled examples and vocabulary the model has not seen. Third, cost at very high volume, where a small fine-tuned model replaces a large general one for a narrow task.
Notice what these have in common: a large number of examples of the desired behaviour already exists. If you would have to create the training data from scratch, the case is much weaker.
The order to do things in
- Prompt properly first, with clear instructions and a few examples. This solves more than people expect.
- Add retrieval if the failures are knowledge failures.
- Improve chunking, indexing and re-ranking before doing anything more exotic.
- Only then consider fine-tuning, and only for behaviour that survived all of the above.
Most projects stop at step three, correctly. The teams that jump to step four first are usually solving a problem they have not yet diagnosed.