Why RAG? LLM Knowledge Limitations

Understand why plain LLMs fail on recent, private and specific facts, and how RAG compares with fine-tuning and long context windows.

35 minBeginnerCode Examples

The Core Idea: A language model is a frozen snapshot of the text it was trained on. It cannot know what happened after training, it cannot see your private files, and it sometimes fills gaps with confident guesses. RAG exists because these four limits are impossible to remove by making the model bigger, but easy to work around by letting the model look things up. This lesson explains each limit and shows when RAG, fine-tuning or a long context window is the better tool.

1. Beginner: What an LLM Really Knows

During training, a large language model (LLM) reads an enormous amount of text and adjusts billions of numbers called parameters (also called weights) so that it gets better at predicting the next token. A token is a small piece of text such as a word or part of a word. Whatever the model learns ends up spread across those parameters. This is called parametric knowledge: knowledge stored inside the model itself.
Think of a photograph of a huge library. The photo captures a lot, but it is taken at one moment, it is a compressed copy, and it does not tell you which book each detail came from. Parametric knowledge has exactly these three properties:
Frozen
Nothing changes after training ends. A model never learns from conversations unless it is trained again.
Compressed
Trillions of words are squeezed into a fixed number of parameters. Common patterns survive well; rare details blur.
Unsourced
The model produces text from patterns. It does not keep a list of which document each fact came from.

Key point: A model is excellent at language, reasoning patterns and widely repeated facts. It is unreliable as a database of exact, recent or private facts. RAG moves those facts out of the parameters and into documents that can be searched.

2. Beginner: The Four Limitations RAG Is Built to Fix

Limitation 1 · Knowledge cutoff
Training data stops at some date, called the knowledge cutoff. A model is usually released months after its cutoff and then used for a long time, so there is always a gap between what it knows and the present day. Ask about a product launch, a law change or a score from after the cutoff and the model either admits ignorance or, worse, answers from older information as if nothing changed. RAG fix: retrieve current documents and put them in the prompt.
Limitation 2 · Hallucinations
A hallucination is a fluent, confident answer that is false or unsupported. It happens because the model is built to produce likely text, not to verify facts. When it lacks a detail, the most likely-sounding continuation is often an invented one: a wrong date, a made-up statistic, a source that does not exist. Hallucinations are most common for rare, specific or recent facts. RAG fix: supply the real passage and instruct the model to answer only from it. This reduces hallucinations a lot, but does not eliminate them, because the model can still misread or ignore the context.
Limitation 3 · Private and proprietary data
Your internal wiki, contracts, customer tickets and product manuals were never in the training data, so the model knows nothing about them. RAG fix: keep those documents in your own searchable store and retrieve the relevant pieces per question. Note that retrieved text is still sent to the model inside the prompt, which is why access control and security get their own lessons later in this course.
Limitation 4 · Frequently changing information
Prices, stock levels, policies, schedules and documentation change daily or weekly. Re-training a model for every change is impractical. RAG fix: update the document and the next search returns the new version. No model change is needed.
Here is how the four limits look in a real support chatbot:
Customer asksPlain LLM problemWith RAG
What is the Pro plan price this month?Old price from training dataCurrent price from the pricing page
How long is the hardware warranty?Confident guessExact term from the warranty document
How do I reset my account in the new app?Never saw the new appSteps from the latest help article

3. Intermediate: RAG vs Fine-Tuning

Fine-tuning means continuing the training of an existing model on your own examples, so its parameters shift slightly. A common first reaction is: if the model does not know my data, I should fine-tune it on my data. Sometimes that is right, but fine-tuning and RAG solve different problems.
Fine-tuning is best for behavior
Teaching a tone of voice, an output format such as strict JSON, a specialized task or a domain style. It changes how the model acts, and this change is permanent in the weights.
RAG is best for facts
Supplying specific, current, private knowledge at answer time. It changes what the model reads, and the knowledge stays in editable documents.
QuestionFine-tuningRAG
What changes?The model parametersThe prompt content
Updating a factPrepare data and train againEdit the document
Citing a sourceHard; knowledge is blended into weightsNatural; the passage is known
Per-user access rulesDifficult; everyone shares the same weightsPossible by filtering documents
Teaching style or formatStrongWeak; prompts help only partly
Main extra effortTraining data and computeRetrieval pipeline and index

They are not rivals. Many production systems use both: fine-tune for tone and format, and use RAG for the facts. A useful rule of thumb is fine-tune for how to answer, retrieve for what to answer with.

4. Intermediate: RAG vs Long Context

The context window is the maximum amount of text a model can read in one request. Some modern models accept very large windows, which raises a fair question: why search at all, when you could paste every document into the prompt? For small collections that is a perfectly good idea. For real knowledge bases it runs into four problems.
Size
Company knowledge bases often hold millions of tokens, more than most context windows can hold, and they keep growing.
Cost and speed
You pay for, and wait for, every token you send, on every question. Sending the whole collection each time is slow and expensive.
Reliability
Research has shown that models can use information in the middle of a very long prompt less reliably than information near the start or end.
Control
With one giant prompt it is hard to filter by user permissions, point to a single source, or see why an answer was produced.
A quick calculation shows the scale. Suppose the knowledge base holds 5 million tokens and a typical RAG prompt, with the question and the top few chunks, uses about 1,500 tokens. Then RAG sends roughly 5,000,000÷1,500≈3,3335{,}000{,}000 \div 1{,}500 \approx 3{,}333 times fewer tokens per question. (These are illustrative numbers, but the shape of the result holds for any large collection.)

When long context wins: one document or a small set that fits comfortably, one-off analysis of a whole report, or tasks that really need to read everything. Often best: use RAG to pick the relevant material, then use a generous context window to read more of it.

5. Advanced: Choosing the Right Approach, and What RAG Does Not Fix

Use these questions as a decision guide. Start from the problem, not from the tool.
Your situationBest starting point
Answers must use recent or changing factsRAG
Answers must use private documents and show sourcesRAG
Model must follow a strict style or output formatPrompting first, then fine-tuning
One short document, one-off questionPaste it into the context window
Large changing knowledge base plus a specific brand voiceRAG plus fine-tuning
RAG is powerful, but it is not magic. Three things it does not fix:
Bad retrieval
If the right passage is not found, the model has no evidence. Later lessons on chunking, embeddings and reranking attack this problem.
Bad documents
If the knowledge base is outdated, duplicated or wrong, RAG faithfully retrieves outdated, duplicated or wrong text.
Weak reading
The model may still misread, ignore or over-trust the context. This is why evaluation and citations matter.

Good RAG behavior includes saying I do not know. When retrieval finds nothing relevant, the best answer is an honest refusal, not a guess. The code in section 7 shows a simple way to do this with a minimum similarity score.

6. Visual Explanation: The Knowledge Gap, Four Problems and the Cost of Context

This figure has three panels. Panel A shows the knowledge gap on a timeline. Panel B is a problem explorer: click any problem to see its symptom, cause and fix. Panel C compares how much text each approach must send per question.
Panel A · The Knowledge GapTime runs from left to right
Plain LLM
Training data: known
After cutoff: unknown
LLM with RAG
Training data: known
Retrieved documents fill the gap
PastTraining cutoffToday
Panel B · Problem ExplorerClick a problem to open it
Panel C · Tokens Sent per QuestionIllustrative numbers, bar lengths not to scale
Whole knowledge base in the promptabout 5,000,000 tokens
RAG: question plus top chunksabout 1,500 tokens
Click to see why fewer tokens is not the only benefitExplain ▼
A short, focused prompt is cheaper and faster, and it also gives the model less irrelevant text to be distracted by. It makes it easy to say exactly which passages the answer used, and it lets you filter documents per user before anything reaches the model.
How to read it: in Panel A the green block is the whole idea of RAG, a bridge over the gap the model cannot see. Panel B lists the four problems and their limits, and Panel C shows why the bridge is also cheap.

7. Python Implementation: Stale Memory vs Live Documents

This program builds a pretend closed-book model that only knows a frozen training snapshot, and a small RAG system that searches live documents. Three questions show the three failure types: a stale fact, a wrong guess, and a question nobody has an answer for. The RAG side uses word-count cosine similarity and a minimum score, so when nothing matches well it refuses instead of guessing. The last lines repeat the token calculation from section 4. Everything uses the standard library only.
stale_memory_vs_rag.pyPython 3.10+ · Standard library only
import mathfrom collections import Counter # 1. A pretend LLM: it only knows a frozen snapshot from training timetraining_snapshot = [    ("pro plan", "The Pro plan costs 20 dollars per month."),    ("warranty", "Hardware usually comes with a 12 month warranty."),] def closed_book(question):    for topic, fact in training_snapshot:        if topic in question.lower():            return fact    return "It was probably the manager."   # a confident guess with no evidence # 2. The live knowledge base: new, private and changing informationlive_documents = [    "Update 2026: The Pro plan now costs 25 dollars per month.",    "Internal policy: employees get 5 extra vacation days after 3 years.",    "Warranty: all hardware products include a 24 month warranty.",] stop_words = set(["the", "is", "a", "of", "how", "much", "does", "per", "who", "to"]) def vectorize(text):    cleaned = "".join(ch for ch in text.lower() if ch.isalnum() or ch == " ")    words = [w for w in cleaned.split() if w not in stop_words]    return Counter(words) def cosine(a, b):    dot = sum(a[w] * b[w] for w in set(a) & set(b))    norm_a = math.sqrt(sum(v * v for v in a.values()))    norm_b = math.sqrt(sum(v * v for v in b.values()))    if norm_a == 0 or norm_b == 0:        return 0.0    return dot / (norm_a * norm_b) # 3. RAG: retrieve the best document, or refuse if nothing matches welldef rag_answer(question, docs, min_score=0.2):    q = vectorize(question)    best_score, best_doc = max((cosine(q, vectorize(d)), d) for d in docs)    if best_score < min_score:        return "I could not find this in the documents."    return best_doc questions = [    "How much does the Pro plan cost per month?",    "How long is the hardware warranty?",    "Who won the office chess tournament?",] for q in questions:    print("Question:", q)    print("  Without RAG:", closed_book(q))    print("  With RAG:   ", rag_answer(q, live_documents))    print("") # 4. The cost of pasting everything into the prompt insteadkb_tokens = 5000000rag_tokens = 1500print("Tokens if the whole knowledge base is pasted in:", kb_tokens)print("Tokens with RAG (question + top chunks):", rag_tokens)print("Reduction factor:", round(kb_tokens / rag_tokens))
Expected output
Question: How much does the Pro plan cost per month?  Without RAG: The Pro plan costs 20 dollars per month.  With RAG:    Update 2026: The Pro plan now costs 25 dollars per month. Question: How long is the hardware warranty?  Without RAG: Hardware usually comes with a 12 month warranty.  With RAG:    Warranty: all hardware products include a 24 month warranty. Question: Who won the office chess tournament?  Without RAG: It was probably the manager.  With RAG:    I could not find this in the documents. Tokens if the whole knowledge base is pasted in: 5000000Tokens with RAG (question + top chunks): 1500Reduction factor: 3333
Read the three results in order. The first shows a stale fact being replaced by an updated document. The second shows a plausible but wrong memory being replaced by the exact term. The third shows the most important RAG habit: when the documents do not contain the answer, say so. The score threshold of 0.2 is a simple stand-in for the more careful relevance checks you will meet in later lessons.

Knowledge Check

Q1. A model was trained with data up to a certain date. Which problem does this create?
Q2. You want a chatbot to always reply in a strict company tone and JSON format. Which tool is the best first fit?
Q3. Why not simply paste a 5-million-token knowledge base into every prompt?

Key Points

✓
An LLM stores knowledge in its parameters: frozen at training time, compressed, and without sources.
✓
Four limits motivate RAG: knowledge cutoff, hallucinations, private data and frequently changing information.
✓
RAG reduces hallucinations by supplying evidence, but it cannot eliminate them.
✓
Fine-tuning changes behavior (style, format, skills); RAG supplies facts. Many systems use both.
✓
Long context is great for small inputs, but for large knowledge bases RAG wins on size, cost, speed and control.
✓
RAG does not fix bad retrieval, bad documents or a model that misreads context.
✓
A good RAG system says it could not find the answer instead of guessing.

The Big Picture

Every limit in this lesson comes from the same root: facts stored in frozen parameters. RAG changes where facts live. They move into documents you control, which can be updated, filtered, cited and searched. Fine-tuning and long context remain useful tools, but for knowledge that is recent, private, large or changing, retrieval is usually the best foundation.
Frozen memory + Live documents = Current, grounded, traceable answers

Up Next: RAG Architecture and the End-to-End Pipeline. You will see every component in one picture: the offline indexing pipeline, the online query pipeline, and how retrieval, context and generation connect.