RAG or Fine-Tuning? A Decision Guide

The question comes up in almost every first conversation about adding AI to a product: should we fine-tune a model on our data, or retrieve from it? They are not alternatives to the same problem, and choosing by cost or fashion rather than by what each technique does leads to expensive detours.

They solve different problems

Retrieval changes what the model knows at answer time. Fine-tuning changes how the model behaves.

If your requirement is "answer questions about our documentation", that is a knowledge problem, and retrieval is the tool. If your requirement is "always reply in this structure, in this register, with these conventions", that is a behaviour problem, and fine-tuning is the tool.

Most teams discover they have a knowledge problem wearing a behaviour problem's clothes.

Reach for retrieval when

• The information changes. Anything that updates weekly is a poor candidate for baking into weights.
• You need citations. A fine-tuned model cannot tell you where an answer came from; a retrieval system can.
• Access control matters. Retrieval can filter by permission at query time. Weights cannot be un-trained per user.
• The corpus is large. You cannot fine-tune your way to recalling ten thousand documents reliably.
• You need to delete something. Removing a document from an index is immediate. Removing it from weights means retraining.

Reach for fine-tuning when

• Output format must be exact and prompting keeps drifting.
• You have a specialised task with abundant examples — classification, extraction, structured transformation.
• Latency or cost demands a smaller model that cannot follow long instructions reliably.
• Tone and house style matter enough that describing them in a prompt wastes tokens on every call.

The option most teams skip

Before either: try a better prompt and better examples. Few-shot prompting with well-chosen examples closes a surprising amount of the gap that people assume requires fine-tuning. It is reversible in minutes, costs nothing to try, and the examples you gather become your fine-tuning dataset later if you still need one.

The sequence that wastes least time is usually: prompt well, add retrieval if the problem is knowledge, add fine-tuning only if behaviour is still wrong after both.

Combining them

They compose. A fine-tuned model that reliably produces your output structure, fed by a retrieval system that supplies current facts, is a sound architecture. The mistake is using fine-tuning to teach facts and retrieval to teach format — each doing the job the other is better at.

What fine-tuning actually costs

Not the training run. The dataset. A fine-tune is only as good as the examples behind it, and assembling a few thousand high-quality, consistently formatted examples is weeks of careful work, not an afternoon. Budget for the labelling, the review, and the second pass when you discover the first thousand examples were inconsistent.

Then budget for doing it again. Fine-tunes go stale as the base model you built on is deprecated, and as your own requirements move.

A test that settles it quickly

Write down ten questions the system must answer correctly. Paste the relevant source material directly into the prompt by hand and ask the model.

If it answers well with the material in front of it, you have a retrieval problem — the model is capable, it just needs the right context delivered automatically.

If it answers badly even with perfect context, retrieval will not save you, and fine-tuning or a stronger model is the honest next step.

That experiment takes an hour and routinely saves months.