Fine-tuning, prompting or retrieval: a decision guide

These three get discussed as alternatives on a cost-and-effort scale. They are not alternatives. They change different things.

What each one actually does

Prompting shapes how the model handles a single request. It is the cheapest to change and it changes nothing durable.

Retrieval puts knowledge the model does not have into the context at query time. It adds facts.

Fine-tuning adjusts the model's weights so it behaves differently by default. It changes behaviour: format, tone, task-specific judgement, domain conventions.

The most common mistake is trying to add knowledge by fine-tuning. It sort of works, and it works badly: the knowledge is baked in at training time, becomes stale immediately, cannot be updated without retraining, and cannot be cited. Then somebody asks where a claim came from and there is no answer.

Use retrieval when

• The knowledge changes. Prices, policies, inventory, case history.
• The answer must cite a source.
• Different users may see different subsets of it.
• The corpus is large relative to what fits in context.

Essentially every enterprise question about company-specific facts falls here.

Use fine-tuning when

• You need a consistent output format that prompting achieves unreliably.
• The task has domain conventions that are hard to express as instructions — how a claims adjuster phrases a reserve rationale, how a radiologist structures an impression.
• Latency or cost require a smaller model to do a job a large one does well. This is the most underrated use: fine-tune a small model on the large model's outputs for a narrow task and serve it for a fraction of the cost.
• You have hundreds to thousands of high-quality examples of the behaviour you want.

Use prompting when

• You are still learning what good output looks like.
• The requirement varies per request.
• The volume does not justify anything more.

Prompting is also the correct first attempt at everything. If it works, stop.

They compose

The mature configuration is usually all three: a fine-tuned model that behaves the way the domain expects, retrieval supplying current governed facts, and prompting handling per-request variation.

Order of adoption: prompt first, add retrieval when facts are needed, fine-tune when behaviour is the remaining gap.

Costs

Prompting: near zero to change, but longer prompts are billed on every request forever. A 2,000-token system prompt at scale is a real line item.

Retrieval: the index, the embedding refresh, the per-request retrieval compute, and the larger contexts. Grows with corpus size, not just request volume.

Fine-tuning: the training run, the data preparation — usually the largest part — the evaluation, and a retraining cadence, because a tuned model drifts as the domain moves.

A caution about data preparation

Fine-tuning quality is dominated by dataset quality, not by hyperparameters. Five hundred carefully curated examples routinely beat fifty thousand scraped ones.

If your fine-tune underperforms, the answer is almost always in the data. Teams spend weeks on learning rates when the actual issue is that 15% of the training examples are wrong.