The Core Idea: A notebook prototype and a production RAG service differ in everything except the idea. Production adds services and APIs, scaling, continuous ingestion, versioning, and failure handling. The questions change from does it work to does it keep working: when a document changes, when the database is slow, when traffic doubles, or when a dependency is down.
1. Intermediate: The Backend at a Glance
A production RAG system is a small set of cooperating services, split along the offline and online lines you learned in the architecture lesson.
| Component | Role | Runs |
|---|---|---|
| Ingestion pipeline | Loads, cleans, chunks and embeds documents | Offline, continuous or scheduled |
| Queue and workers | Spread embedding work and absorb spikes | Offline |
| Vector database and metadata store | Hold vectors, chunk text and filters | Both |
| Query API and orchestrator | Run the transformation, retrieval and generation steps | Online |
| Model gateway | Calls the language model with limits and retries | Online |
| Cache and observability | Speed up repeated work and show what happens | Both |
The query API is usually a simple contract: it receives the question, the user identity and optional filters, and returns the answer, the citations and a trace id for debugging. Keep services stateless where you can, so that more copies can run when load grows.
2. Intermediate: Scaling the Vector Database
Vector search is memory and compute hungry. Plan capacity before it surprises you. A first estimate of raw vector memory is the number of vectors times the dimension times the bytes per number, plus index overhead that depends on the index type.
Replicas
Copies of the index that answer queries in parallel. They raise read throughput and availability.
Sharding
Splitting the data across machines when it no longer fits on one. Queries are sent to all shards and the results are merged.
Managed or self-hosted
A managed service removes operations work. Self-hosting gives control over data and cost, but needs skills and on-call time.
Also plan for: backups, index rebuild time, and what happens to latency while a rebuild or a heavy ingestion is running. Separate heavy writes from live reads where possible.
3. Intermediate: Batch, Real-Time and Incremental Ingestion
How new knowledge reaches the index is a design choice with three common answers.
| Mode | How it works | Good for | Downside |
|---|---|---|---|
| Batch | Re-process everything or a large set on a schedule | Static or slowly changing sources | Stale between runs, wasteful |
| Real-time (event driven) | Process each document change as it happens | Fast-changing data | More moving parts |
| Incremental | Detect what changed with fingerprints and process only that | Almost everything | Needs reliable change detection |
Make it safe to repeat: ingestion jobs fail and restart. Make them idempotent, so running the same job twice gives the same result, and send repeatedly failing documents to a separate queue for inspection instead of blocking everything.
4. Intermediate: Updates, Deletions, Freshness and Versioned Knowledge Bases
Updates must replace the old chunks of a document, not add to them, or both versions will be retrieved. Deletions must remove every chunk of the document, which matters for compliance when content has to disappear on request.
Freshness is the delay between a change in the source and the new content being searchable. Define a target, such as new documents searchable within 15 minutes, measure it, and alert when it is missed.
Snapshot. Treat each published state of the knowledge base as a named version.
Build beside. Create a new index version next to the live one instead of editing the live index.
Test. Run the golden set against the new version.
Switch. Point the service at the new version, for example by moving an alias.
Roll back. Keep the previous version available so you can switch back in seconds.
5. Advanced: Reliability and Fallback Strategies
Every dependency can fail: the vector database, the embedding service, the language model. A resilient system expects this and degrades gracefully instead of showing an error.
| Technique | What it does |
|---|---|
| Timeouts | Never wait forever for a slow dependency |
| Retries with backoff | Try again after growing delays, for transient errors |
| Circuit breaker | Stop calling a failing service for a while so it can recover |
| Rate limits and quotas | Protect your service and your bill |
| Fallback chain | Primary retrieval, then keyword search, then a cached answer, then an honest message |
Decide ahead of time: what is acceptable when something fails. A slightly less relevant answer from keyword search is usually better than an error page, but for high-stakes domains an honest we cannot answer right now may be better than a degraded guess.
6. Visual Explanation: The Production Map
Panel A draws the offline and online paths. Panel B shows what the ingestion code below decided for each document. Panel C is a failure explorer.
Panel A · Two Paths, One StoreHeavy work stays offline
Sources→Queue→Parse and chunk→Embed workers→Vector database
↑ the same store is read by the online path ↓
User→Query API→Retrieve→Model gateway→Answer with citations
Panel B · Incremental IngestionFrom the code in section 7
| Document | Day 40 source | Action |
|---|---|---|
| a | unchanged | skip |
| b | text changed | update to version 2 |
| c | gone from the source | delete |
| d | new | add |
Document a is flagged stale only because it is 39 days old, which exceeds the 30-day freshness rule
Panel C · Failure ExplorerClick a failure to open it
Click to see why the new index is built beside the old one▼
Editing the live index risks serving half-updated results. Building a separate version lets you test it and switch in one step, and gives you an instant rollback if something is wrong.
How to read it: Panel A is the map, Panel B shows the ingestion logic at work, and Panel C turns common incidents into prepared responses.
7. Python Implementation: Incremental Ingestion, Snapshots, Freshness and Fallback
The program keeps a small index of document ids with fingerprint, version and update day. An ingestion function compares each incoming document with the index and decides to add, update, skip or delete. After each run a snapshot of the index is saved. A freshness report flags documents older than 30 days, a rollback restores an earlier snapshot, and a fallback chain serves a query from keyword search when the primary retrieval raises a timeout.
production_patterns.pyPython 3.10+ · Standard library only
import hashlib def fingerprint(text): return hashlib.sha256(text.encode("utf-8")).hexdigest()[:6] index = dict()snapshots = dict() def ingest(source_docs, today): actions = [] for doc_id, text in source_docs: fp = fingerprint(text) if doc_id not in index: index[doc_id] = (fp, 1, today) actions.append("add " + doc_id) elif index[doc_id][0] != fp: index[doc_id] = (fp, index[doc_id][1] + 1, today) actions.append("update " + doc_id) else: actions.append("skip " + doc_id) present = set(doc_id for doc_id, text in source_docs) for doc_id in list(index): if doc_id not in present: del index[doc_id] actions.append("delete " + doc_id) return actions def take_snapshot(name): snapshots[name] = dict(index) print("Day 1 :", ingest([("a", "refund policy v1"), ("b", "shipping times"), ("c", "old faq")], 1))take_snapshot("kb-1")print("Day 40:", ingest([("a", "refund policy v1"), ("b", "shipping times updated"), ("d", "new support guide")], 40))take_snapshot("kb-2") print("")for doc_id in sorted(index): fp, version, updated = index[doc_id] age = 40 - updated flag = "STALE" if age > 30 else "fresh" print("doc", doc_id, "| version", version, "| age", age, "days |", flag) print("")print("Snapshots:", sorted(snapshots), "| kb-1 had", len(snapshots["kb-1"]), "docs, kb-2 has", len(snapshots["kb-2"]))index.clear()index.update(snapshots["kb-1"])print("Rolled back to kb-1, live index now has:", sorted(index)) print("")def primary_search(query): raise TimeoutError("vector database timed out") def keyword_search(query): return ["keyword result for: " + query] def answer(query): for name, search in [("primary", primary_search), ("keyword fallback", keyword_search)]: try: return name, search(query) except Exception as error: print(" ", name, "failed:", error) return "none", ["Sorry, the knowledge base is unavailable."] print("Served by:", answer("refund policy"))Expected output
Day 1 : ['add a', 'add b', 'add c']Day 40: ['skip a', 'update b', 'add d', 'delete c'] doc a | version 1 | age 39 days | STALEdoc b | version 2 | age 0 days | freshdoc d | version 1 | age 0 days | fresh Snapshots: ['kb-1', 'kb-2'] | kb-1 had 3 docs, kb-2 has 3Rolled back to kb-1, live index now has: ['a', 'b', 'c'] primary failed: vector database timed outServed by: ('keyword fallback', ['keyword result for: refund policy'])On day 40 the unchanged document is skipped without any embedding work, the changed one becomes version 2, the missing one is deleted and the new one is added, which is exactly the incremental behavior described in the lesson. The freshness report flags the document that has not changed for 39 days; whether that matters depends on the document type. The rollback restores the earlier snapshot in one step. Finally, the primary search fails with a timeout and the user is still served by the fallback. Real systems add retries, metrics and persistent storage, but the decisions are the same.
Knowledge Check
Q1. Why should ingestion jobs be idempotent?
Q2. What is the safest way to deploy a rebuilt knowledge base?
Q3. The vector database times out. What should a resilient system do?
Key Points
✓
A production RAG backend separates offline ingestion from the online query path and uses stateless services where possible.
✓
Scale the vector database with replicas for reads and shards for size, and plan for memory, backups and rebuilds.
✓
Incremental ingestion with fingerprints processes only what changed; make jobs idempotent and use a dead-letter queue.
✓
Updates must replace old chunks and deletions must remove every chunk of a document.
✓
Set and monitor a freshness target for how quickly changes become searchable.
✓
Version the knowledge base: build beside, test, switch with an alias, and keep a rollback.
✓
Plan fallbacks: timeouts, retries with backoff, circuit breakers and a degrade-gracefully chain.
The Big Picture
Production RAG is mostly the discipline of keeping a good prototype good over time. The same retrieval and generation ideas apply, wrapped in services that can scale, update safely, be rolled back and survive failures of their own parts.
Ingest incrementally → Version the knowledge base → Serve reliably → Fall back gracefully
Up Next: RAG Observability, Security and Governance. You will learn how to log, trace and monitor a RAG system, defend against prompt injection and data leakage, and keep a trustworthy record of what the system knows and who can see it.