The Core Idea: Real knowledge is not only paragraphs. It lives in scanned PDFs, tables, screenshots, diagrams, source code and recorded meetings. RAG can handle all of them with one idea: turn each kind of content into searchable records that keep a pointer back to the original, either by converting it to text or by embedding it directly. This lesson shows how each format is handled and how to combine them.
1. Intermediate: Two Strategies for Non-Text Content
There are two ways to make an image, a table or a recording searchable.
Convert to text
Run OCR on scans, write captions for images, transcribe audio, turn tables into sentences. Then index the text like any other. Simple and works with any text retriever.
Embed natively
Use a multimodal embedding model that places images and text in one shared space, so a text query can retrieve a picture directly. Keeps visual detail that captions miss.
Common practice: start with conversion to text, which is cheaper and easier to debug. Add native image embeddings when captions lose information your users ask about. Either way, store the original file and a locator with every record.
2. Intermediate: PDF and Table RAG
PDF RAG depends on extraction quality, as in the document processing lesson: layout-aware parsing, repeated headers removed, OCR for scanned pages, and the page number stored with every chunk so citations can point to it.
Table RAG needs extra care, because a table is meaningful only as a whole grid. Three common techniques are used:
| Technique | How | Best for |
|---|---|---|
| Row sentences | Turn each row into a sentence that repeats the column names | Lookup questions about single rows |
| Markdown or HTML table | Keep the small table intact in one chunk | Small tables the model can read as a grid |
| Table summary | Index a text summary and keep the full table linked to it | Large tables and numeric analysis |
For calculations: if the question needs sums or comparisons across many rows, load the table into a database and use the SQL approach from the previous lesson.
3. Intermediate: Image and Multimodal RAG
Image RAG retrieves pictures, charts and diagrams. With the convert-to-text strategy, each image gets a caption and any text read from it, and those are searched. With native embeddings, the image vector is compared to the query vector directly.
Multimodal RAG then sends the retrieved images, together with text, to a model that can read images, so it can answer questions about a chart or a screenshot. The same discipline applies: the answer should cite the image file, and a person should be able to open it.
Watch out: captions are written by a model and can be wrong or vague. Review a sample, and keep captions specific, for example bar chart of quarterly revenue with the Q3 growth value, not just a chart.
4. Intermediate: Code RAG and Audio RAG
Code RAG treats a function or class, not a fixed number of lines, as a chunk, because a cut through the middle of a function is meaningless. A syntax tree gives the exact boundaries. Each chunk should include the file path, the signature and the docstring, since developers search by what code does, and those parts describe it.
Audio RAG begins with speech-to-text. The transcript is chunked by time or by topic, and each chunk keeps its start time and, when available, the speaker. Answers can then link to the exact moment of a recording. Transcription errors on names and technical words are common, so spell-check against a glossary when possible.
| Content | Natural chunk | Locator to keep |
|---|---|---|
| Code | Function or class | File path and line range |
| Audio | A few sentences or a topic span | Timestamp and speaker |
| Image | One image with caption and text | File name or URL |
| Section or paragraph | Page number | |
| Table | Row or whole small table | Table id and row |
5. Advanced: Mixed-Format Knowledge Bases
A real company knowledge base holds all of the above. Two designs are common.
One unified index
Every record has a modality tag, a locator and text. One search covers everything and ranks across formats. Simple, but one embedding model must suit all content.
Separate indexes per modality
Each type gets the best model and settings. A router, or a fusion step, combines the results. More flexible and more work.
Define a shared record shape: modality, locator, text, metadata and permissions.
Process each format with its best tool, as in the document processing lesson.
Evaluate separately per modality, since an average hides a weak format.
Cite with the locator so users can open the exact page, timestamp, file or row.
6. Visual Explanation: Every Format Becomes a Record with a Locator
Panel A shows the conversion step for each format. Panel B lists the results of a search across mixed content from the code below. Panel C is a format explorer.
Panel A · Format to RecordEverything ends up searchable
Documents
PDF and tables
Parse layout, OCR scans
↓
Rows to sentences
↓
Text with page and row
Media
Images and audio
Captions and text from images
↓
Speech to text with timestamps
↓
Text with file and time
Code
Source files
Parse into functions
↓
Keep name and docstring
↓
Text with path and name
Panel B · One Search, Four Kinds of ContentFrom the code in section 7
| Question | Found in | Locator |
|---|---|---|
| refund amount with tax | code | function refund_total |
| when do they mention free shipping | transcript | at 00:45 |
| show me revenue growth | image | chart_q3.png |
| how many users in the Pro plan | table | row Pro |
Panel C · Format ExplorerClick a format to open it
Click to see why every record keeps a locator▼
A locator such as a page, timestamp or function name lets the answer point to the exact place in the original, which makes citations and verification possible in every format.
How to read it: Panel A shows how each format is converted, Panel B proves a single search can cover them, and Panel C lists what each format needs.
7. Python Implementation: One Index for Code, Audio, Images and Tables
The program builds a single list of records from four kinds of content. Python code is split into functions with the standard ast module, and each function becomes a record from its name and docstring. An audio transcript becomes records with time stamps. Two images are represented by captions. Table rows become sentences. Every record has a modality, a locator and text, so one simple search can rank across all of them.
mixed_format_rag.pyPython 3.10+ · Standard library only
import ast stop_words = set(["the", "a", "of", "how", "is", "do", "does", "in", "to", "what", "for", "and", "show", "me"]) def words(text): cleaned = "".join(ch if ch.isalnum() else " " for ch in text.lower()).split() return set(w[:-1] if w.endswith("s") and len(w) > 3 else w for w in cleaned if w not in stop_words) records = [] # 1. Code: one chunk per function, found with the ast modulesource = chr(10).join([ "def refund_total(price, tax):", ' """Return the refund amount including tax."""', " return price + tax", "", "def shipping_cost(weight):", ' """Return the shipping fee for a parcel weight in kilograms."""', " return 5 + 2 * weight",])for node in ast.parse(source).body: if isinstance(node, ast.FunctionDef): doc = ast.get_docstring(node) records.append(("code", "function " + node.name, node.name.replace(chr(95), " ") + " " + doc)) # 2. Audio transcript: chunks keep their timestampstranscript = [(0, "Welcome to the product demo."), (45, "Premium members get free shipping on every order."), (95, "Refunds take five business days to arrive.")]for start, text in transcript: stamp = str(start // 60).zfill(2) + ":" + str(start % 60).zfill(2) records.append(("transcript", "at " + stamp, text)) # 3. Images: searched through captions or text read from the imageimages = [("chart_q3.png", "bar chart showing quarterly revenue growth of 12 percent"), ("diagram_flow.png", "diagram of the order shipping workflow")]for name, caption in images: records.append(("image", name, caption)) # 4. Table rows turned into sentencestable = [("Basic", "10", "5 users"), ("Pro", "25", "50 users")]for plan, price, limit in table: records.append(("table", "row " + plan, "The " + plan + " plan costs " + price + " dollars and allows " + limit + ".")) def search(query): q = words(query) return max(records, key=lambda r: len(q & words(r[2]))) print("Indexed", len(records), "records from 4 kinds of content")for query in ["refund amount with tax", "when do they mention free shipping", "show me revenue growth", "how many users in the Pro plan"]: kind, where, text = search(query) print(query.ljust(36), "->", kind.ljust(10), where)Expected output
Indexed 9 records from 4 kinds of contentrefund amount with tax -> code function refund_totalwhen do they mention free shipping -> transcript at 00:45show me revenue growth -> image chart_q3.pnghow many users in the Pro plan -> table row ProEach question lands on a different kind of content, and every result carries a locator that a user interface could turn into a link: a function name, a time in the recording, an image file or a table row. The search itself is word matching here. In a real system the text would be embedded, images might also get native image embeddings, and speech and OCR models would produce the text, but the record shape would stay the same.
Knowledge Check
Q1. Why is a code function a better chunk than a fixed number of lines?
Q2. What must every record keep to support citations across formats?
Q3. A question needs the sum of a large table column. What is the best approach?
Key Points
✓
Non-text content becomes searchable by converting it to text or by embedding it natively.
✓
PDF RAG depends on extraction quality, page numbers and OCR for scans.
✓
Tables work as row sentences, intact small tables or summaries linked to the full table; use SQL for calculations.
✓
Image RAG uses captions and extracted text or multimodal embeddings; captions must be specific and checked.
✓
Code is chunked by function or class with path, signature and docstring; audio by time with timestamps and speakers.
✓
Mixed knowledge bases need a shared record shape with modality, locator, text, metadata and permissions.
✓
Evaluate each modality separately, and cite with the locator.
The Big Picture
The architecture of RAG does not change when the content does. Only the front door changes: each format needs its own parser, converter or embedding step, and a locator that leads back to the original. After that, retrieval, context building and citation work as before.
Any format → Searchable record with locator → Retrieve → Ground → Cite with the locator
Up Next: RAG Evaluation. You are entering the Evaluation phase. You will learn retrieval metrics, generation metrics, golden datasets, LLM-as-a-judge and failure analysis, so you can prove that your RAG system actually works.