The Core Idea: Once every chunk has an embedding, you need somewhere to keep millions of vectors and a way to find the nearest ones in milliseconds. A vector database stores vectors together with their text and metadata, and builds an index that makes nearest-neighbor search fast. The central idea of this lesson is a trade: accept a tiny chance of missing the best match in exchange for searching a small fraction of the data.
1. Beginner: What a Vector Store Keeps and Does
A vector store is built around one record type. Every record holds four things, and all four are needed by RAG.
Id
A unique key, such as a chunk id, used to update or delete the record.
Vector
The embedding, which is what similarity search runs on.
Text
The original chunk, which is what the model finally reads.
Metadata
Source, date, version, permissions and other fields for filtering and citations.
The basic operations are simple: upsert (insert or update a record), query (return the nearest records to a vector, optionally with a metadata filter), and delete.
2. Beginner: Vector Stores vs Vector Databases
The words are often used loosely, but a useful distinction exists. A vector store or library is code that runs inside your own application and mainly handles storage and search. A vector database is a full database system built around vectors.
| Aspect | Library or embedded store | Vector database |
|---|---|---|
| Runs | Inside your application process | As a service, self-hosted or managed |
| Persistence and backups | You handle it | Built in |
| Many users and updates at once | Limited | Designed for it |
| Scaling and access control | Mostly manual | Provided |
| Best for | Prototypes and small apps | Production systems |
3. Intermediate: Exact Search and the Scale Problem
The simplest search is brute force, also called a flat index: compare the query with every stored vector and keep the best. It is exact, easy to build and perfectly fine for small collections. The cost grows linearly. With vectors of dimension , one query needs about multiplications.
For 10 million vectors of 768 dimensions that is operations for a single query. With many users asking at once, that is too slow and too expensive, so large systems search only part of the data.
Do not over-engineer: for a few hundred thousand chunks or fewer, exact search is often fast enough, and it never misses the true best match. Reach for approximate indexes when measurements show you need them.
4. Intermediate: Approximate Nearest Neighbor Search
Approximate nearest neighbor (ANN) indexes organize vectors so a query can skip most of them. They occasionally miss a true neighbor, so quality is measured by recall, the share of the exact top results that the index finds:
| Index family | Main idea | Typical trade-off |
|---|---|---|
| Cluster based (IVF) | Group vectors into clusters, search only the closest few clusters | Fast and compact; recall depends on how many clusters you probe |
| Graph based (HNSW) | Link each vector to near neighbors and walk the graph toward the query | Very good recall and speed; uses more memory |
| Compression (quantization) | Store shortened or lower precision vectors | Saves memory; slight loss of accuracy |
These ideas are often combined, for example clusters plus compression. The tuning knobs always trade recall against speed and memory.
5. Advanced: Tools, Tuning and Filtering
You will meet many options. They differ mainly in how they are run, not in the basic idea.
| Tool | Kind | Good starting point for |
|---|---|---|
| FAISS | Open-source similarity search library | Fast in-process search and research |
| Chroma | Open-source embedding database, easy to embed in an app | Prototypes and small projects |
| Pinecone | Managed vector database service | Teams that want no infrastructure work |
| Weaviate | Open-source vector database | Search combined with rich filtering |
| Milvus | Open-source vector database built for large scale | Very large collections |
Every index has parameters. For cluster indexes you choose how many clusters exist and how many are probed per query. For graph indexes you choose how many links each node keeps and how widely the search explores. More effort per query means higher recall and higher latency.
Remember filtering and updates: metadata filters interact with the index, since filtering after a search can leave too few results, and some index types are slow to update or delete. Check how your chosen database handles both before you commit.
6. Visual Explanation: Searching Everything vs Searching Part
Panel A compares exact and approximate search. Panel B walks through a cluster-based search. Panel C is an explorer of the tools from the table.
Panel A · Exact vs ApproximateSame question, different effort
Exact (flat) search
Checks every vector
Compare with all N vectors
↓
Sort by similarity
↓
Perfect recall, slow at scale
Approximate (ANN) search
Checks a small part
Jump to the most promising region
↓
Compare with only those vectors
↓
Tiny recall loss, much faster
Panel B · A Cluster SearchIVF in four moves
Query vector→Find nearest clusters→Scan only their members→Top-k results
Probing more clusters raises recall and cost together
Panel C · Tool ExplorerClick a tool to open it
Click to see why recall is never exactly guaranteed▼
An approximate index decides where to look using shortcuts. If the true nearest neighbor sits in a region the shortcut skips, it is missed. Tuning raises the chance of finding it but never makes it zero.
How to read it: Panel A is the trade, Panel B is one way to make it, and Panel C is the landscape of tools that implement it.
7. Python Implementation: Exact Search vs a Cluster Index
The program generates 2,000 vectors in 20 natural groups, then builds a simple cluster index: it picks 20 centroids, runs three rounds of re-centering, and stores which vectors belong to each cluster. A query is answered by probing only the closest few clusters. For 50 test queries it compares the approximate top 5 with the exact top 5 and reports recall and the number of distance checks.
ivf_search_demo.pyPython 3.10+ · Standard library only
import random random.seed(7)DIM = 8N = 2000K = 20 centers = [[random.random() for j in range(DIM)] for i in range(K)]vectors = []for i in range(N): c = centers[i % K] vectors.append([x + (random.random() - 0.5) * 0.3 for x in c]) def dist2(a, b): return sum((x - y) * (x - y) for x, y in zip(a, b)) centroids = [vectors[i * (N // K)] for i in range(K)] def assign(): lists = [[] for c in range(K)] for idx, v in enumerate(vectors): nearest = min(range(K), key=lambda c: dist2(v, centroids[c])) lists[nearest].append(idx) return lists for step in range(3): lists = assign() for c in range(K): if lists[c]: centroids[c] = [sum(vectors[i][j] for i in lists[c]) / len(lists[c]) for j in range(DIM)]lists = assign() def exact_search(q, k): return sorted(range(N), key=lambda i: dist2(q, vectors[i]))[:k] def ivf_search(q, k, nprobe): order = sorted(range(K), key=lambda c: dist2(q, centroids[c]))[:nprobe] candidates = [i for c in order for i in lists[c]] best = sorted(candidates, key=lambda i: dist2(q, vectors[i]))[:k] return best, K + len(candidates) queries = []for i in range(50): c = centers[random.randrange(K)] queries.append([x + (random.random() - 0.5) * 0.3 for x in c]) print("Vectors:", N, "| lists:", K, "| exact search checks", N, "vectors per query")for nprobe in (1, 2, 4, 8): recalls = [] work = [] for q in queries: truth = set(exact_search(q, 5)) found, cost = ivf_search(q, 5, nprobe) recalls.append(len(truth & set(found)) / 5) work.append(cost) print("nprobe", nprobe, "| recall@5", round(sum(recalls) / len(recalls), 2), "| checks per query", round(sum(work) / len(work)))Expected output
Vectors: 2000 | lists: 20 | exact search checks 2000 vectors per querynprobe 1 | recall@5 0.96 | checks per query 177nprobe 2 | recall@5 0.99 | checks per query 302nprobe 4 | recall@5 1.0 | checks per query 523nprobe 8 | recall@5 1.0 | checks per query 780Probing one cluster out of twenty already finds almost every true neighbor while checking under a tenth of the vectors, and probing four finds all of them with about a quarter of the work. These numbers are so good because the synthetic data is clearly clustered. Real embeddings are less tidy, so expect to probe more clusters for the same recall, and always measure recall on your own data.
Knowledge Check
Q1. What does an approximate nearest neighbor index give up?
Q2. Why can exact search be a good choice for a small knowledge base?
Q3. In a cluster index, what happens when you probe more clusters per query?
Key Points
✓
A vector store record holds an id, a vector, the original text and metadata.
✓
A library runs inside your app; a vector database adds persistence, scaling, access control and concurrency.
✓
Exact search compares the query with every vector, so cost grows with the size of the collection.
✓
Approximate indexes such as IVF and HNSW search only part of the data and are measured by recall.
✓
Compression such as quantization saves memory at a small cost in accuracy.
✓
FAISS, Chroma, Pinecone, Weaviate and Milvus differ mostly in how they are run and scaled.
✓
Check how filtering, updates and deletes behave before choosing an index.
The Big Picture
With a vector database you can store the knowledge base and search it at the speed users expect. The index is a performance tool: it never makes results more relevant, it only finds the nearest vectors faster. Relevance still depends on chunks, embeddings and everything that follows.
Chunks → Embeddings → Index in a vector database → Fast nearest-neighbor search
Up Next: Retrieval Fundamentals. The Embeddings phase is complete. Next you start the Retrieval phase: what retrieval returns, how top-k works, and how recall and precision describe its quality.