Data Cleaning and Metadata

Learn how to clean and normalize extracted text, remove duplicates, and design the metadata, document IDs, timestamps and versioning that keep a RAG knowledge base trustworthy.

35 minBeginnerCode Examples

The Core Idea: Text that was extracted from documents is rarely ready to search. It carries stray spaces, page numbers, odd characters, repeated copies and several versions of the same file. Cleaning makes the text consistent, deduplication removes repeats, and metadata records where each piece came from, when, and which version it is. Together they decide whether your knowledge base is trustworthy or full of noise and contradictions.

1. Beginner: Why Clean Data Matters

Imagine a library where the same book appears five times, some pages have the page number printed in the middle of sentences, and old editions sit next to new ones with no labels. A librarian searching it would hand you three copies of the same outdated page. A RAG retriever behaves exactly like that librarian: it returns whatever the index contains, in whatever condition it is in.
Noise
Page numbers, headers and odd symbols take up space in chunks and dilute their meaning.
Repetition
Copies of the same text fill the top results, pushing out other useful passages.
Contradiction
An old and a new version of a policy both get retrieved, and the model may quote the wrong one.

Key point: retrieval cannot tell which passage is current, clean or authoritative unless you tell it. Cleaning and metadata are how you tell it.

2. Beginner: Cleaning and Normalization

Cleaning removes what should not be there. Normalization makes equivalent things look identical. Both aim at one goal: the same idea should always be written the same way.
OperationExampleBe careful
Collapse whitespaceMany spaces and line breaks become single spacesKeep paragraph breaks if chunking will use them
Normalize charactersNon-breaking spaces and fancy variants become plain charactersKeep accents that change meaning
Remove boilerplateLines such as Page 3 of 10, repeated headers, cookie noticesDo not delete real content that looks similar
Fix encoding errorsGarbled symbols replaced by the intended charactersFix at the source if you can
Standardize formatsDates and units written one consistent wayNever change the meaning of a number

Less is more: modern embedding models understand normal text well. You usually do not need to lowercase everything, remove punctuation or strip common words. Clean away real noise, but keep the natural wording, because aggressive cleaning can destroy meaning.

3. Intermediate: Duplicate Removal

Duplicates enter a knowledge base in many ways: the same file saved in two folders, an email thread quoted in every reply, a web page reachable through several URLs. They hurt in three ways: they fill the top-k results with identical text, they waste embedding and storage cost, and they inflate how important a passage looks.
Exact duplicates
Compute a hash, a short fingerprint of the cleaned text. Two texts with the same hash are identical, and the second can be skipped. This is fast and cheap, and it works best after cleaning, because two copies that differ only in spacing become identical.
Near duplicates
Texts that are almost the same, such as a document with one changed sentence. Compare the overlap of their word sets, or use similarity search, and treat pairs above a chosen threshold as duplicates for review.
A simple overlap measure is the Jaccard similarity of two word sets AA and BB:
J(A,B)=∣A∩B∣∣A∪B∣J(A,B) = \dfrac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}
If two passages each use 8 distinct words and share 7 of them, the union has 8+8−7=98 + 8 - 7 = 9 words, so J=7/9≈0.78J = 7/9 \approx 0.78. A value of 1 means identical word sets and 0 means nothing in common.

Not every repeat is a mistake. Legal boilerplate or a standard disclaimer may legitimately appear in many documents. Decide per source type whether repeated text should be removed, kept once, or kept everywhere with its metadata.

4. Intermediate: Metadata, Document IDs and Timestamps

Metadata is data about the data. It does not go into the embedding, but it travels with every chunk and unlocks features later. Plan your fields early, because adding them afterwards means reprocessing everything.
FieldExampleWhat it enables
Document IDdoc-5e15a5Stable identity, updates and deletions
Sourcepolicy.txt or a page URLCitations and trust
Timestamp2025-06-15Freshness filters and ordering
Version and statusv2, activeHide outdated versions
Fingerprinta73e2f47Detecting duplicates and changes
Category or ownerlegal, HR, supportFiltering and access rules
Notice that the document ID and the fingerprint are different things. The document ID is a stable name for the document, derived for example from its source path, so it stays the same across versions. The fingerprint is a hash of the content, so it changes whenever the text changes. Using both lets you say this is the same document, but its content is new.

5. Advanced: Document Versioning and Keeping the Index Fresh

Documents change. A good pipeline treats every incoming file as one of four cases, decided by comparing its document ID and fingerprint with what is already stored:
New ID. A brand-new document. Process it fully and mark it active as version 1.
Same ID, same fingerprint. Nothing changed. Skip it and save the embedding cost.
Same ID, new fingerprint. The content was updated. Create the next version, mark the old one superseded, and re-embed the new text.
Same fingerprint, different ID. A copy of an existing document. Skip it as a duplicate.
Marking the old version as superseded matters as much as adding the new one. If the old chunks stay searchable, retrieval can return both the 30-day and the 60-day return policy, and the model has no way to know which is current. Most vector stores let you filter on a status field, or delete the old chunks outright.

Deletion and audit: when a document is withdrawn, its chunks must disappear from the index too, otherwise answers keep quoting content that no longer exists. Keeping a history of versions with timestamps also gives you an audit trail: you can explain what the system knew at any past date.

Processing only what changed is called incremental ingestion. It keeps the cost of freshness low, and later production lessons build on it.

6. Visual Explanation: From Messy Text to a Versioned Catalog

Panel A shows a cleaning diff: red lines are removed or changed, green lines are what remains. Panel B replays four documents arriving over time and shows what the pipeline decides for each. Panel C is the metadata card that finally sits beside the text.
Panel A · Cleaning DiffBefore and after for one document
− Refund Policy
+ Refund Policy
− Returns accepted within [odd space] 30 days.
+ Returns accepted within 30 days.
− Page 1 of 2
Keep your receipt.
Panel B · Four ArrivalsTime runs from top to bottom
2025-01-10policy.txtkept · v1 active
2025-02-01policy_copy.txtskipped · same fingerprint
2025-03-05shipping.txtkept · v1 active
2025-06-15policy.txtnew content · v2 active, v1 superseded
Panel C · The Metadata CardTravels with every chunk
Metadata
doc id: doc-5e15a5
source: policy.txt
date: 2025-06-15 · version 2
status: active
Text that gets embedded
Refund Policy Returns accepted within 60 days. Keep your receipt.
Click to see why the metadata is not embeddedExplain ▼
Embeddings should capture what the text means. IDs, dates and status flags are better used for exact filtering, such as only active documents or only items from this year, which a vector comparison cannot do reliably.
How to read it: Panel A cleans the text, Panel B decides what to keep over time, and Panel C shows the finished record that retrieval will actually search and filter.

7. Python Implementation: Clean, Deduplicate and Version Documents

The program below processes the same four documents as Panel B. It cleans each one with Unicode normalization, whitespace collapsing and boilerplate removal, fingerprints the result with a SHA-256 hash, skips exact duplicates, and gives each source a stable document ID and a version number. When a new version of a source arrives, the old one is marked superseded. Only the standard library is used.
clean_dedupe_version.pyPython 3.10+ · Standard library only
import hashlibimport unicodedata # (source, date, lines). chr(160) is a non-breaking space, a common extraction artifactraw_documents = [    ("policy.txt", "2025-01-10", ["Refund   Policy", "Returns accepted within" + chr(160) + "30 days.", "Page 1 of 2", "Keep your receipt."]),    ("policy_copy.txt", "2025-02-01", ["Refund Policy", "Returns accepted within 30 days.", "Page 1 of 2", "Keep your receipt."]),    ("shipping.txt", "2025-03-05", ["Shipping Info", "Delivery takes 5 days.", "Page 1 of 1"]),    ("policy.txt", "2025-06-15", ["Refund Policy", "Returns accepted within 60 days.", "Page 1 of 2", "Keep your receipt."]),] # 1. Cleaningdef is_boilerplate(line):    lowered = line.lower()    return lowered.startswith("page ") and " of " in lowered def clean(lines):    kept = []    for line in lines:        line = unicodedata.normalize("NFKC", line)        line = " ".join(line.split())        if line and not is_boilerplate(line):            kept.append(line)    return " ".join(kept) # 2. Fingerprints and IDsdef short_hash(text):    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:8] # 3. The catalog: (doc_id, source, version, date, status, fingerprint, text)catalog = []version_of = dict()seen = set() for source, date, lines in raw_documents:    text = clean(lines)    fingerprint = short_hash(text.lower())    if fingerprint in seen:        print("SKIP duplicate:", source, "->", fingerprint)        continue    seen.add(fingerprint)    doc_id = "doc-" + short_hash(source)[:6]    version = version_of.get(source, 0) + 1    version_of[source] = version    for i, old in enumerate(catalog):        if old[1] == source and old[4] == "active":            catalog[i] = old[:4] + ("superseded",) + old[5:]    catalog.append((doc_id, source, version, date, "active", fingerprint, text)) print("")for doc_id, source, version, date, status, fingerprint, text in catalog:    print(doc_id, source.ljust(13), "v" + str(version), date, status.ljust(10), fingerprint) print("")print("Searchable text:")for doc_id, source, version, date, status, fingerprint, text in catalog:    if status == "active":        print("-", source, "|", text)
Expected output
SKIP duplicate: policy_copy.txt -> 846d9d7d doc-5e15a5 policy.txt    v1 2025-01-10 superseded 846d9d7ddoc-fb5015 shipping.txt  v1 2025-03-05 active     c0064e9fdoc-5e15a5 policy.txt    v2 2025-06-15 active     a73e2f47 Searchable text:- shipping.txt | Shipping Info Delivery takes 5 days.- policy.txt | Refund Policy Returns accepted within 60 days. Keep your receipt.
Three things happened. The copy was caught because, after cleaning, its text matched the first file exactly, even though the raw lines differed in spacing. Both policy files share one document ID, because the ID comes from the source name, while their fingerprints differ because the content changed. And only the newest policy version is left active, so a search filtered on active status can never return the outdated 30-day rule.

Knowledge Check

Q1. Why is it better to clean text before computing a duplicate fingerprint?
Q2. A document is re-uploaded with the same ID but a new fingerprint. What should the pipeline do?
Q3. Which of these is best stored as metadata instead of inside the embedded text?

Key Points

✓
Cleaning removes noise; normalization makes equivalent text look identical.
✓
Do not over-clean: keep natural wording, because aggressive cleaning can damage meaning.
✓
Exact duplicates are found with content hashes; near duplicates with overlap measures such as Jaccard similarity.
✓
Metadata should include document ID, source, timestamp, version, status and ownership or category.
✓
A stable document ID names the document; a fingerprint detects when its content changes.
✓
On every update, supersede or delete old chunks, or retrieval will mix old and new facts.
✓
Process only changed documents (incremental ingestion) to keep freshness cheap.

The Big Picture

At this point your documents are clean, unique and labeled. That is the real foundation of a good knowledge base. Every later improvement, from better chunking to reranking, works on top of it, and none of them can repair text that is noisy, duplicated or out of date.
Extract → Clean → Deduplicate → Add metadata → Version → Ready to chunk

Up Next: Chunking Strategies. With clean, labeled documents in hand, you will learn how to split them into the right-sized pieces, with fixed-size, recursive, sentence, paragraph and semantic chunking, and how overlap and chunk size affect retrieval.