llm architecture

Take a support team that ships two prototypes on the same base model. The first searches the product documentation on every question and pastes the top passages into the prompt. The second was fine-tuned on 4,000 past tickets. In the demo, the fine-tuned one sounds better: right terminology, exact escalation format. Three weeks later pricing changes. The fine-tuned model keeps quoting the old plan; the retrieval system picks up the new page the morning it’s published.

Neither prototype failed at what it was built to do. Retrieval-augmented generation (RAG) changes what the model can see when it answers. Fine-tuning changes how the model behaves. So the RAG vs fine-tuning decision starts with a plain question: does your problem live in the facts, in the form of the output, or in both?

The short answer: start with RAG when answers depend on knowledge that changes, is large, or must be cited. Investigate fine-tuning when the model already knows enough but won’t reliably produce the format, style or task behaviour you need, and prompting hasn’t fixed it. Combine them only when you’ve measured both problems at once.

Key Takeaways

• RAG updates knowledge at query time without touching weights. Fine-tuning is a weak, risky way to add facts: research found RAG consistently beat it for knowledge injection.
• Fine-tuning earns its place on behaviour: output format, tone, classification and repeated specialised tasks, especially when it lets you shorten prompts.
• RAG mostly costs you per query (retrieval plus extra input tokens); fine-tuning mostly costs you per update (data, training, evaluation, regression testing). Query volume and update frequency decide which is cheaper.
• More retrieved text isn’t better. Models use information in the middle of long contexts less reliably, so retrieval precision beats retrieval volume.
• Neither approach is a security boundary. Retrieved documents can carry injected instructions; training data can be poisoned.
• A fine-tuned model is tied to its base model’s lifecycle. On OpenAI’s platform, tuned models are retired along with their base, so budget for re-tuning.

---

Quick Navigation

---

RAG vs Fine-Tuning: Two Different Ways to Change an AI System

The original RAG paper by Lewis and colleagues framed the distinction precisely. A model’s weights are parametric memory: knowledge absorbed in training, hard to inspect and hard to update. A searchable document index is non-parametric memory: knowledge the model reads at inference time. The authors pointed to two problems with parametric memory that retrieval helps with, providing provenance for an answer and updating what the model knows.

In a RAG system, the model doesn’t change. At query time the application embeds the question, searches an index, optionally reranks, and inserts the best passages into the prompt. Re-index an edited document and the next answer can reflect it.

Fine-tuning runs a training job on examples of the input and output you want, producing a modified set of weights or a small set of adapter weights. Parameter-efficient methods such as LoRA train a tiny fraction of the parameters (up to 10,000 times fewer than full fine-tuning of GPT-3 175B, the paper reports) without adding inference latency. That makes fine-tuning cheaper. It doesn’t make it a reliable way to store facts.

Five things get lumped together as “teaching the model”:

What needs to change	Example	Usual first tool
Knowledge	Current refund policy, a product’s API limits	RAG
Behaviour	Ask a clarifying question before acting	Prompting, then fine-tuning
Style	House tone, terminology, reading level	Prompting, then fine-tuning
Task format	Fixed JSON schema, ticket triage labels	Structured outputs or prompting, then fine-tuning
Tool-use pattern	Which tool to call, in what order	Prompting and tool definitions, then fine-tuning

Only the first row is about facts. The rest are about form, where fine-tuning has a real case.

RAG vs Fine-Tuning: What Changes When the Source of Truth Changes?

Most enterprise knowledge moves. Product documentation changes each release, policies are revised quarterly, manuals get errata, and support knowledge bases are edited daily. What matters is how often facts change and how quickly a wrong answer hurts.

RAG handles change by re-indexing, but it isn’t instant or automatic. A document edited at 09:00 isn’t retrievable until the ingestion pipeline has chunked, embedded and written it to the index. If the old version isn’t deleted, both can be retrieved and blended. Two sources that disagree, such as a policy page and an outdated FAQ, produce an answer that cites one and contradicts the other. These are data-engineering problems: freshness targets, deletion handling, versioning and source precedence.

Fine-tuning handles change by retraining, and that’s the core reason it’s rarely the first choice for factual knowledge. Ovadia and colleagues compared the two for knowledge injection and found that RAG consistently outperformed unsupervised fine-tuning, for knowledge the model had seen before and for entirely new facts. They also found models struggled to learn new facts this way unless exposed to many variations of each one. A separate study by Gekhman and colleagues found that fine-tuning examples containing new knowledge are learned much more slowly than familiar ones, and that once learned they increase the model’s tendency to hallucinate. Their conclusion: models acquire facts mainly in pre-training, and fine-tuning mostly teaches them to use what they already know.

That doesn’t rule fine-tuning out of fast-changing domains. A claims assistant might retrieve current policy wording while a fine-tuned model handles the stable part: extracting fields into a fixed schema and flagging exceptions consistently. The facts live outside the model; the behaviour lives inside it.

RAG vs Fine-Tuning Cost: The Cost of a Query Versus the Cost of an Update

RAG is cheap to change and pays on every query. Fine-tuning pays each time something changes and can be cheaper per query.

• RAG costs fall into fixed and variable parts. Fixed: preparing documents, building the chunking and embedding pipeline, search infrastructure and re-indexing. Variable, per query: embedding the question, the search, any reranking and, usually the largest line, the extra input tokens from retrieved passages. Evaluation and monitoring sit on top.
• Fine-tuning costs are weighted the other way. Fixed: curating training examples, training jobs, experiments, evaluation against a baseline and, on every refresh, retraining, regression testing, versioning and rollback. Variable: inference on the tuned model, which may be priced differently from the base model or need dedicated serving if self-hosted.

One fixed cost is easy to miss: a fine-tune is bound to its base model. OpenAI’s fine-tuning guide, checked on 25 October 2026, says the platform is being wound down and closed to new users; existing customers can train until 6 January 2027, and tuned models last only as long as their base model. Whatever your provider, budget for re-tuning and re-evaluating when the base model retires.

A break-even model

Define, over one evaluation period:

• Q = number of queries
• C_R = fixed RAG setup and maintenance cost
• V_R = average variable RAG cost per query
• C_F = fixed fine-tuning, evaluation and maintenance cost
• V_F = average variable cost per query on the fine-tuned model

Then:

Total RAG cost = C_R + Q × V_R

Total fine-tuning cost = C_F + Q × V_F

When fine-tuning has the higher fixed cost (C_F > C_R) and the lower per-query cost (V_F < V_R), the volume at which the two totals are equal is:

Q* = (C_F − C_R) / (V_R − V_F)

Below Q*, RAG is cheaper; above it, fine-tuning is. If V_F isn’t lower than V_R, for instance because a tuned model’s per-token premium outweighs the context saved, fine-tuning never breaks even on cost.

Most of the gap V_R − V_F usually comes from context. If fine-tuning lets you drop T_ctx retrieved tokens from every prompt at an input price of P_in per token, then V_R − V_F is roughly T_ctx × P_in plus the retrieval cost, minus any premium the tuned model charges per token. OpenAI’s optimisation guide makes the same point from the other side: fine-tuning can allow shorter prompts with fewer examples and less context, saving tokens at scale. That saving only exists if the facts you were retrieving can be dropped, which is rarely true for changing knowledge.

Three assumptions limit the model. It’s only valid if both systems meet the same quality, latency, security and availability targets. C_F recurs with every data refresh, so frequent updates multiply it. And it ignores the cost of being wrong. Add an error term, with p as the share of answers that are wrong and K as the average cost of a wrong answer (a refund, an escalation, a compliance review), and each total gains Q × p × K. A cheaper system with a higher error rate can cost more overall. UniverseBlend’s breakdown of what inference actually costs per token covers how request shape, caching and batching change V in practice.

Context Length and Inference Latency in RAG vs Fine-Tuning

Every retrieved passage is input the model must process before it writes the first token. That prefill work grows with prompt length, so retrieval adds to time-to-first-token as well as cost. Generation speed changes less with context length than the wait before it starts.

More context also isn’t more accuracy. Liu and colleagues found that models used relevant information best when it sat at the beginning or end of the input and noticeably worse when it sat in the middle, even for models built for long contexts. Retrieving the top 20 chunks “to be safe” can push the one that matters into exactly that position.

The levers are about precision: chunks sized to the unit of meaning (a policy clause, an API endpoint), a reranker that cuts a wide search to a few passages, and compression of long sources. UniverseBlend’s piece on context engineering in AI agents covers how ordering and caching stable prompt sections cut cost and latency, which applies directly to RAG prompts.

Fine-tuning doesn’t make context unnecessary. A tuned model still needs today’s facts, the user’s details and any document it’s asked about. What it can remove is scaffolding: long instructions and few-shot examples that taught the format. That saving is real at high volume, and different from dropping retrieved knowledge.

RAG vs Fine-Tuning: Accuracy, Evaluation and Failure Modes

The two architectures fail in recognisably different ways, and a fluent answer can hide either.

RAG fails when retrieval misses the right passage, the index holds stale or wrong documents, chunk boundaries split the sentence that mattered, the reranker drops key evidence, or the model answers beyond what the text supports. The last is hardest to spot, because the answer arrives with citations.

Fine-tuning fails when training examples are inconsistent (the model learns the errors), when it overfits and handles new inputs worse, when it regresses on abilities the base model had, and when teams assume training on a document means the model will recall it. The Gekhman results suggest that assumption can worsen hallucination.

Evaluate both on a representative test set drawn from real queries, not on demo prompts. For RAG, measure retrieval separately from generation: were the right passages retrieved (recall at k), was the set focused, and was the answer faithful to it? The Ragas framework evaluates these dimensions without human labels for every case. For both, measure correctness, task success, latency and cost per query, and rerun a regression suite on every index rebuild or model version.

RAG vs Fine-Tuning: Security and AI Agent Implications

Retrieval puts external text into the model’s context, and the model can’t reliably tell data from instructions. Greshake and colleagues showed attackers could plant prompts in content an application was likely to retrieve and steer its behaviour and tool calls. In an agent, a poisoned wiki page is an attack path.

Fine-tuning carries a different risk: a model trained on poisoned data carries the problem into every answer, and editing a document won’t fix it. The OWASP Top 10 for LLM Applications 2025 lists both, as prompt injection (LLM01) and data and model poisoning (LLM04), alongside vector and embedding weaknesses (LLM08), which covers retrieval stores themselves.

Neither architecture is a security boundary. Fine-tuning doesn’t reliably train away injection, and phrase filters are easy to evade. The controls that hold sit outside the model: permission-aware retrieval so users only retrieve what they’re allowed to read, provenance on every indexed document, access controls on the index, least-privilege tool permissions and confirmation for irreversible actions. UniverseBlend’s analysis of how agent prompt-injection benchmarks score failures explains why passing a benchmark isn’t proof of safety.

The Facts–Form–Frequency Test: A RAG vs Fine-Tuning Decision Framework

The Facts–Form–Frequency test, developed for this article rather than an industry standard, splits each requirement in two and asks how often each part changes.

• Facts: what the model must know that it doesn’t reliably know now.
• Form: how the output must look or behave, consistently.
• Frequency: how often each changes, and how many queries you’ll serve between changes.

The rules: changing facts go to retrieval. Form that prompting can’t hold goes to fine-tuning, once measured. Frequency and volume decide whether the fixed cost pays.

Scenario	Recommendation	Assumption behind it
Frequently changing factual knowledge	RAG	Updates must appear within hours, not training cycles
Stable domain-specific output format	Fine-tuning, after prompting and structured outputs fall short	The format is consistent and examples are available
Repeated specialised task behaviour	Fine-tuning	High volume, stable task, measurable gain over prompting
Large collections of reference documents	RAG	Corpus too large for the context window and changing
Strict source attribution	RAG	Answers must cite the passage they used
High-volume, predictable queries	Evaluate fine-tuning on cost	Q exceeds Q* at equal quality
Current facts and consistent behaviour	Both	Both gaps measured separately
Base model already meets the bar	Neither	Prompting plus evals pass

When a Hybrid RAG and Fine-Tuning Architecture Makes Sense

A hybrid uses each method for what it’s good at. A fine-tuned extraction model returns contracts in a fixed schema while retrieval supplies the current clause library to check them against.

Hybrids cost more than either alone: you pay C_R and C_F, and a regression could come from the index or the model. Justify one with two measurements, a behaviour gap prompting didn’t close and a knowledge gap the tuned model alone didn’t close. Otherwise start with the simplest architecture that meets requirements, usually a strong base model with good prompting and retrieval.

RAG vs Fine-Tuning: A Workload-Based Implementation Checklist

A planning meeting should be able to answer these before committing:

1. Baseline: What does the base model score on your test set with good prompting alone?
2. Test data: Do you have a few hundred real, labelled queries, including hard and adversarial ones?
3. Update frequency: How often do the underlying facts change, and how stale may an answer be?
4. Volume: What’s Q per month now and in a year, and what are C and V for each option?
5. Cost: Are you logging input tokens, retrieval calls and cost per successful task?
6. Latency: What time-to-first-token and end-to-end targets must hold at peak?
7. Security: Who may retrieve which documents, which tools can the model call, and what needs human confirmation?
8. Rollback: Can you revert an index build or model version in minutes?
9. Lifecycle: When does your base model retire, and who re-tunes and re-tests on its replacement?
10. Monitoring: Which drop in recall, faithfulness, task success or cost triggers review?

Conclusion

Start with RAG when answers depend on changing facts, a large corpus or a citation requirement. Investigate fine-tuning when the base model knows enough but won’t hold a format, style or task behaviour at your volume after proper prompting. Use both only when you’ve measured both gaps.

The useful next step is small: take 200 real queries, run them through the base model with good prompting, and label every failure as a facts problem or a form problem. The split tells you which approach to build first.

FAQ

Is RAG better than fine-tuning?

Neither in general. RAG suits current or large bodies of knowledge; fine-tuning suits consistent behaviour and format.

Can RAG and fine-tuning be used together?

Yes: a fine-tuned model for task format plus retrieval for current facts. It’s worth the extra cost only when both gaps are measured.

Is RAG cheaper than fine-tuning?

Not always. RAG has lower fixed costs but adds cost to every query; fine-tuning can win at high volume if it removes context, but every update adds fixed cost.

Does fine-tuning add new knowledge to an LLM?

Unreliably. Models learn new facts slowly this way, and learning them can increase hallucination.

Does RAG increase token usage?

Yes. Retrieved passages are extra input tokens on every query. Tighter retrieval and reranking reduce the increase.

Which approach is better for frequently changing information?

RAG. Updating an index takes minutes to hours; updating weights takes a training cycle plus evaluation.

Can fine-tuning improve structured output?

Often. Try structured-output features and prompting first, then fine-tune if format errors persist at your volume.

Does RAG make an AI agent more secure?

No. Retrieved documents can carry prompt injections. Security comes from permissions, provenance and tool controls outside the model.

---

Keep reading

Here are the latest posts from the blog.