The Core Idea: Plain chunking treats a document as a flat stream of words. Real documents have structure: titles, sections, lists, tables and links between parts. Advanced chunking uses that structure so the retriever can find small precise pieces while the model still receives enough surrounding context to answer correctly.
1. Intermediate: The Small-to-Big Problem
Retrieval and generation want opposite things. Retrieval works best with small, focused chunks, because a short chunk produces a sharp embedding. Generation works best with larger passages, because the model needs surrounding sentences to answer fully and correctly.
The fix is to separate the unit you search from the unit you read: search over small pieces, then hand the model a larger piece around the match. This idea is often called small-to-big retrieval.
Search small
Index short chunks, such as single sentences or short paragraphs, for precise matching.
Read big
After a match, expand to the surrounding paragraph or section before building the prompt.
Link them
Store an id on every small chunk that points to its larger parent so the expansion is a quick lookup.
2. Intermediate: Parent-Child and Hierarchical Chunks
In parent-child chunking every small child chunk keeps the id of the larger parent chunk it was cut from. Only children are embedded and searched. When a child matches, the system returns its parent.
Hierarchical chunking extends this to many levels: document, section, paragraph, sentence. Matches can be expanded one level or several, depending on how much context the question needs.
| Expansion method | What the model receives | Best when |
|---|---|---|
| Parent chunk | The whole parent of the matching child | Documents have clear sections |
| Sentence window | The matching sentence plus a few neighbors on each side | Dense prose without headings |
| Auto-merge | The parent, but only if many of its children matched | Questions that span a whole section |
Watch the budget: bigger context can bury the answer and use up prompt space. Expand only as far as needed, and merge duplicates when several children share one parent.
3. Intermediate: Structure-Aware Chunking
Many documents announce their own structure through headings: Markdown headers, HTML heading tags, numbered sections in PDFs. Structure-aware chunking splits at those boundaries so a chunk never mixes two sections.
A powerful trick is to keep the heading path with each chunk, for example Returns, then Refunds. The path can be stored as metadata and also placed at the start of the chunk text. A sentence like They take 5 business days becomes self-explanatory once it begins with the heading it belongs to, and the embedding captures that topic too.
Rule of thumb: split on the biggest structural boundary available first, and fall back to size-based splitting only for sections that are still too long.
4. Intermediate: Tables, Lists and Special Content
Content that is not ordinary prose needs its own rules, because cutting it carelessly destroys its meaning.
Tables
Never cut a row in half. For small tables keep the whole table in one chunk. For large ones turn each row into a line that repeats the column names, such as Plan Basic, Price 10.
Lists
Keep a list together with the sentence that introduces it. A bullet list without its lead-in often has no meaning.
Code and figures
Keep code blocks whole and attach captions or alt text to figures, so the surrounding explanation travels with them.
Check by reading: print a few table and list chunks and ask whether a person could answer a question from that chunk alone. If not, the chunking needs more context.
5. Advanced: Relationships Between Chunks
Chunks are not independent. Sections refer to each other, a summary points at the details, and versions replace older ones. Recording these links lets the retriever follow them.
| Link | Stored as | Used for |
|---|---|---|
| Parent | A parent id on each child | Small-to-big expansion |
| Neighbors | Previous and next chunk ids | Sentence windows and reading on |
| References | Ids of sections that are mentioned, such as see section 4 | Following citations inside documents |
| Versions | Document id plus version and status | Hiding outdated chunks |
Each extra link adds indexing work and more ways for the system to go wrong, for example expanding too far and flooding the prompt. Add relationships when evaluation shows that single chunks are missing context, not before.
6. Visual Explanation: Search Small, Read Big
Panel A shows the hierarchy of a document. Panel B contrasts the unit that is searched with the unit that is read. Panel C is an explorer of special content types.
Panel A · The HierarchyEach level can be a chunk size
Document→Sections→Paragraphs→Sentences
Search at the lowest level, read at a higher level
Panel B · Child Searched, Parent ReadTwo jobs, two sizes
Child chunk
Used for search
Short and focused
↓
Sharp embedding
↓
Finds the exact match
Parent chunk
Used for reading
Whole section with heading
↓
Full context
↓
Goes into the prompt
Panel C · Special ContentClick a type to open it
Click to see the cost of bigger context▼
Returning a parent is safer for answers, but it uses more prompt space and can include unrelated text. Measure whether the extra context really improves answers before expanding everywhere.
How to read it: Panel A names the levels, Panel B assigns each a job, and Panel C lists the traps that special content sets for a careless splitter.
7. Python Implementation: Header Paths, Parent-Child Search and Table Rows
The program splits a small Markdown document into sections and keeps the heading path. Every sentence becomes a child chunk that remembers its parent section. A search matches the best child, then returns the whole parent section for the model to read. The last lines turn table rows into self-contained chunks that repeat their column names.
structure_chunking.pyPython 3.10+ · Standard library only
doc_lines = [ "# Returns", "## Refunds", "Refunds go to the original payment method. They take 5 business days.", "## Exchanges", "Exchanges are free within 30 days. Items must be unused.", "# Shipping", "## Delivery", "Standard delivery takes 5 days. Express delivery takes 2 days.",] def words(text): cleaned = "".join(ch for ch in text.lower() if ch.isalnum() or ch == " ") return set(cleaned.split()) def split_sections(lines): sections = [] h1 = "" h2 = "" for line in lines: if line.startswith("## "): h2 = line[3:] sections.append([h1 + " > " + h2, ""]) elif line.startswith("# "): h1 = line[2:] else: sections[-1][1] = (sections[-1][1] + " " + line).strip() return sections def make_children(sections): children = [] for parent_id, (path, text) in enumerate(sections): for sentence in text.split("."): if sentence.strip(): children.append((parent_id, path + ": " + sentence.strip() + ".")) return children def search(question, children, sections): q = words(question) best = max(children, key=lambda child: len(q & words(child[1]))) parent_path, parent_text = sections[best[0]] return best[1], parent_path + ": " + parent_text sections = split_sections(doc_lines)children = make_children(sections)print("Sections:", len(sections), "| child chunks:", len(children)) question = "How long do refunds take?"child, parent = search(question, children, sections)print("Question:", question)print("Child (searched):", child)print("Parent (sent to the model):", parent) def table_row_chunks(header, rows): return [" | ".join(h + ": " + v for h, v in zip(header, row)) for row in rows] print("")for line in table_row_chunks(["Plan", "Price", "Users"], [["Basic", "10", "5"], ["Pro", "25", "50"]]): print(line)Expected output
Sections: 3 | child chunks: 6Question: How long do refunds take?Child (searched): Returns > Refunds: They take 5 business days.Parent (sent to the model): Returns > Refunds: Refunds go to the original payment method. They take 5 business days. Plan: Basic | Price: 10 | Users: 5Plan: Pro | Price: 25 | Users: 50The question matched one precise sentence, yet the model would receive the full Refunds section with its heading path. Notice that the heading path appears inside every child, so even a bare sentence stays understandable. The table rows each carry their column names, so a retrieved row still makes sense on its own.
Knowledge Check
Q1. In parent-child chunking, which chunks are embedded and searched?
Q2. Why keep the heading path with each chunk?
Q3. How should a large table be chunked?
Key Points
✓
Retrieval prefers small chunks and generation prefers large ones, so separate the search unit from the reading unit.
✓
Parent-child chunking searches children and returns their parent section.
✓
Hierarchical chunking supports several levels, from document down to sentence.
✓
Structure-aware chunking splits at headings and keeps the heading path with each chunk.
✓
Tables, lists and code need special rules so their meaning survives splitting.
✓
Store links such as parent id, neighbors, references and versions, but add them only when evaluation shows a need.
✓
Expanding context too far wastes prompt space, so measure the benefit.
The Big Picture
Good chunking respects how people wrote the document. By searching small pieces and reading larger ones, and by keeping headings and relationships, you give retrieval precision and give the model context in the same system.
Detect structure → Cut at boundaries → Link children to parents → Search small → Read big
Up Next: Embeddings for RAG. With the data side complete, you move to the representation side: what embeddings are, how meaning becomes numbers, and why similar texts end up close together.