Introduction to RAG

Learn what Retrieval-Augmented Generation is, how it differs from a traditional LLM, why it exists, and build a tiny RAG pipeline from scratch in Python.

30 minBeginnerCode Examples

The Core Idea: A language model only knows what it learned during training. Retrieval-Augmented Generation (RAG) gives it an open book: before answering, the system searches your documents, finds the most relevant passages, and hands them to the model together with the question. The model then writes an answer based on that evidence instead of relying on memory alone.

1. Beginner: The Open-Book Exam

Picture two students taking the same exam about your company. The first student studied months ago and must answer purely from memory. The second student is allowed to bring the company handbook and look things up before writing each answer. Both are equally smart, but the second one will be more accurate, especially about details that are new, rare or private.
Closed-book student = a plain LLM
A large language model (LLM) is an AI system trained on huge amounts of text to predict and generate language. Everything it knows is frozen inside its parameters at training time.
Open-book student = a RAG system
The same model, plus a search step that fetches relevant pages from a document collection and places them in front of the model before it answers.

Key point: RAG does not make the model smarter and does not change its parameters. It changes what the model gets to read at the moment it answers.

2. Beginner: What RAG Is in Three Steps

The name RAG describes the three things that happen every time a user asks a question:
R · Retrieve
Search a collection of documents and pick the few passages most related to the question.
A · Augment
Build a new prompt that contains the question and the retrieved passages as context.
G · Generate
The LLM reads that prompt and writes an answer grounded in the context.
Here is a tiny example. A customer asks: What is your refund policy? A plain LLM has never seen your policy and might guess a common one. A RAG system first retrieves the passage Returns are accepted within 30 days of purchase, adds it to the prompt, and the model answers with the real policy.

Analogy: a librarian (the retriever) fetches the right books from the shelves, and a writer (the LLM) reads those books and composes the answer. Neither could do the whole job as well alone.

3. Beginner: RAG vs a Traditional LLM

A traditional LLM call is simple: question in, answer out. A RAG call adds a retrieval step in the middle. This table shows what changes.
QuestionTraditional LLMRAG system
Where does knowledge come from?Training data onlyTraining data plus retrieved documents
Can it use private documents?No, it never saw themYes, if they are in the knowledge base
How do you update its knowledge?Retrain or fine-tune the modelAdd or edit documents
Can it show its sources?Not reliablyYes, it can point to the passages used
What extra parts are needed?NoneDocument store, search, prompt building

Trade-off: RAG buys freshness, privacy and traceability, and pays for it with a more complex system. The next lesson explores exactly why that trade is usually worth it.

4. Intermediate: RAG Terminology

RAG has its own vocabulary. Learn these words now, because every later lesson builds on them.
TermPlain meaning
Knowledge baseThe collection of documents the system can search.
ChunkA small piece of a document, such as a few paragraphs, stored and searched as one unit.
EmbeddingA list of numbers that represents the meaning of a text, so similar meanings get similar numbers.
Vector storeA database that keeps embeddings and can quickly find the closest ones to a query.
QueryThe user question that the system must answer.
RetrieverThe component that finds the most relevant chunks for a query.
Top-kThe number k of best-matching chunks the retriever returns.
ContextThe retrieved text placed inside the prompt for the model to read.
GeneratorThe LLM that writes the final answer from the prompt.
GroundingMaking sure the answer is supported by the retrieved context.
HallucinationA confident-sounding answer that is false or unsupported.
Two phases tie these terms together. In the indexing phase (done ahead of time) documents are split into chunks, converted to embeddings and stored. In the query phase (done for every question) the retriever searches that store, the prompt is built, and the generator answers. Lesson 3 covers this pipeline in detail.

5. Intermediate: Basic Use Cases and Their Limits

RAG fits any situation where answers must come from a specific, changing or private body of text.
Customer support
A chatbot answers from help-center articles, return policies and product manuals.
Document question answering
Ask questions about contracts, reports or research papers instead of reading every page.
Internal knowledge search
Employees ask natural-language questions over wikis, tickets and meeting notes.
Developer assistants
Answers grounded in a project documentation or codebase rather than general memory.
Research helpers
Summaries and answers that point back to the papers they came from.
Regulated domains
Legal, medical or financial answers where citing the exact source matters.

Advanced note: RAG is only as good as its retrieval. There are two main ways it fails. If the retriever misses the right passage, the model has no evidence and may guess. If the retriever finds the right passage but the model ignores or misreads it, the answer is still wrong. Most of the later lessons in this course are about preventing these two failures.

RAG is also not always needed. For general knowledge that the model already handles well, or tiny knowledge sets that fit directly in the prompt, a plain LLM call can be simpler and cheaper.

6. Visual Explanation: Plain LLM vs RAG, Step by Step

The figure has two parts. Part 1 compares what happens to a question in a plain LLM and in a RAG system. Part 2 breaks RAG into its three steps.
Part 1 · Plain LLM vs RAGNotice where the knowledge comes from
Option A · Plain LLMMemory only
User question
↓
LLM answers from training memory
↓
May be outdated or invented
Option B · RAGMemory + documents
User question
↓
Retriever searches your documents
↓
Best passages added to the prompt
↓
Answer grounded in your data
Part 2 · The Three Steps of RAGRetrieve, then Augment, then Generate
Step 1 · Retrieve
Find Evidence
Search the knowledge base and return the top-k most relevant chunks.
Step 2 · Augment
Build the Prompt
Combine the question, the chunks and the instructions into one prompt.
Step 3 · Generate
Write the Answer
The LLM reads the prompt and answers using the supplied context.
Result: fresher, private and traceable answers without retraining
Click to see what happens before any question arrivesExplain ▼
Ahead of time, an indexing step collects the documents, splits them into chunks, converts each chunk to an embedding and stores everything in a searchable index. Step 1 then searches that prepared index, which is why retrieval is fast at question time.
How to read it: compare Option A and Option B first, then zoom into the three steps of Option B. The only new part is the middle: the search over your own documents.

7. Python Implementation: A Tiny RAG Pipeline from Scratch

This program uses only the Python standard library. It stores four short documents, turns each text into word counts, and measures similarity with cosine similarity, which compares two count vectors by the angle between them:
cos⁡(θ)=A⋅B∥A∥ ∥B∥\cos(\theta) = \dfrac{A \cdot B}{\lVert A \rVert \, \lVert B \rVert}
A score of 1 means the texts use the same words in the same proportions, and 0 means they share no words. The program retrieves the top 2 documents for a question and builds the augmented prompt. It stops there, because the final step would be sending that prompt to an LLM. Real systems use learned embeddings instead of word counts, which you will meet in the embeddings lessons.
tiny_rag.pyPython 3.10+ · Standard library only
import mathfrom collections import Counter # 1. The knowledge base (four tiny documents)documents = [    "The company refund policy allows returns within 30 days of purchase.",    "Standard shipping takes 5 to 7 business days inside the country.",    "Premium members get free shipping on every order.",    "Our support team is available Monday to Friday from 9 am to 5 pm.",] def tokenize(text):    cleaned = "".join(ch for ch in text.lower() if ch.isalnum() or ch == " ")    return cleaned.split() def vectorize(text):    return Counter(tokenize(text)) def cosine(a, b):    common = set(a) & set(b)    dot = sum(a[w] * b[w] for w in common)    norm_a = math.sqrt(sum(v * v for v in a.values()))    norm_b = math.sqrt(sum(v * v for v in b.values()))    if norm_a == 0 or norm_b == 0:        return 0.0    return dot / (norm_a * norm_b) # 2. RETRIEVE: score every document and keep the top kdef retrieve(question, docs, k=2):    q = vectorize(question)    scored = [(cosine(q, vectorize(d)), d) for d in docs]    scored.sort(reverse=True)    return scored[:k] # 3. AUGMENT: put the context and the question into one promptdef build_prompt(question, retrieved):    lines = ["Answer using ONLY the context below.", "", "Context:"]    for score, text in retrieved:        lines.append("- " + text)    lines.append("")    lines.append("Question: " + question)    return lines question = "What is the refund policy for returns?"retrieved = retrieve(question, documents, k=2) print("Retrieved passages:")for score, text in retrieved:    print(round(score, 3), "|", text)print("")print("Prompt sent to the LLM:")for line in build_prompt(question, retrieved):    print(line)
Expected output
Retrieved passages:0.456 | The company refund policy allows returns within 30 days of purchase.0.114 | Standard shipping takes 5 to 7 business days inside the country. Prompt sent to the LLM:Answer using ONLY the context below. Context:- The company refund policy allows returns within 30 days of purchase.- Standard shipping takes 5 to 7 business days inside the country. Question: What is the refund policy for returns?
Notice the weak second result. Its score is low and it shares only the word the with the question, yet top-k still returns it. Good retrieval systems learn to rank, filter and sometimes return fewer chunks. The refund passage is the one that actually answers the question, and the model would use it to reply that returns are accepted within 30 days.

Knowledge Check

Q1. What does RAG change about how an LLM answers?
Q2. Your company updates its return policy today. What is the easiest way to make a RAG chatbot know about it?
Q3. The retriever returns the wrong passages. What is the most likely result?

Key Points

✓
RAG stands for Retrieval-Augmented Generation: retrieve evidence, augment the prompt, generate the answer.
✓
A plain LLM answers from frozen training memory; a RAG system also reads documents at answer time.
✓
RAG does not change the model. Knowledge is updated by editing documents, not by retraining.
✓
Core vocabulary: knowledge base, chunk, embedding, vector store, retriever, top-k, context, grounding.
✓
Typical uses: customer support, document question answering, internal search, developer assistants.
✓
Answer quality is limited by retrieval quality: missing or wrong passages lead to weak answers.

The Big Picture

RAG turns a language model from a closed-book student into an open-book one. The model keeps its language skills, and the retrieval system supplies the facts. That split of responsibilities is the foundation of almost every AI assistant that works with company, product or research data.
Question → Retrieve → Augment the prompt → Generate a grounded answer

Up Next: Why RAG? LLM Knowledge Limitations. You will look closely at the problems RAG solves, namely knowledge cutoffs, hallucinations, private data and changing information, and compare RAG with fine-tuning and long context windows.