The Core Idea: The context window is the model's shared working memory: prompt, history and answer all have to fit. Managing that budget well, and understanding the KV cache behind it, keeps conversations fast, cheap and accurate.
- Beginner: What Is a Context Window?
The context window is the maximum amount of text, measured in tokens, that a model can look at in one go. It is the model's working memory. Everything has to fit inside it: your system instructions, the whole conversation so far, any documents you paste in, and the answer the model is about to write.
A token is a piece of a word. A rough rule for English is that 1 token is about 0.75 words, so a 100,000-token window is roughly 75,000 words, about the length of a short novel.
Key point: The window is shared. If your prompt uses 7,000 tokens of an 8,000-token window, the answer can be at most about 1,000 tokens long.
- Beginner: What Happens When the Text Is Too Long?
A model has no memory between requests. A chat application re-sends the whole conversation with every new message. When that history no longer fits, something must be dropped. Applications use a few strategies:
| Strategy | How it works | Downside |
|---|---|---|
| Truncate the oldest | Drop the earliest messages first | Forgets early instructions or facts |
| Summarize | Replace old messages with a short summary | Summaries can lose details |
| Retrieve (RAG) | Store everything outside, fetch only the relevant pieces | Needs a retrieval system |
| Pin important parts | Always keep the system prompt and key facts | Uses part of the budget permanently |
Tip: Always keep the system prompt and the newest message. Drop or compress the middle of the history first.
- Medium: Longer Is Not Always Better
A bigger window is useful, but it is not free and it is not perfect:
Practical rule: Put the most important instructions at the beginning or end, and send only the context you need. A short, focused prompt often beats a huge one.
- Medium: The KV Cache and Prompt Caching
As you saw in the attention lesson, a model stores the keys and values of earlier tokens in a KV cache so it does not recompute them. That cache lives in GPU memory and grows with every token of every active conversation.
Prompt caching is a related service feature. If many requests start with the same long prefix (a big system prompt, a document, a set of examples), the provider can reuse the already-computed cache for that prefix. Cached input tokens are usually billed at a discount and are faster.
Design tip: Put the stable part of your prompt first and the changing part last. Then repeated requests share the same prefix and can hit the cache.
- Advanced: Managing Memory in Long Conversations
Serving many users with long contexts is mainly a memory problem. Common techniques:
| Technique | Idea |
|---|---|
| KV cache quantization | Store the cache in 8-bit or 4-bit numbers instead of 16-bit |
| Grouped-query attention | Share keys and values between heads (smaller cache by design) |
| Paged KV cache | Store the cache in small pages instead of one big block, to waste less memory |
| Prefix caching | Reuse the cache of a shared beginning across requests |
| Sliding window / eviction | Keep only recent tokens, or drop the least useful ones |
- Visual Explanation: The Context Budget
The figure shows who shares the context window and what to do when the conversation no longer fits.
Click to see the budget formulaCompare ▼
Part 1 shows the two sides that compete for the same tokens. Part 2 compares the three common ways to make a long history fit. Open the bar for the budget rule used in the code.
- Python Implementation: A Context-Window Manager
This script estimates token counts, keeps the newest messages that fit in a given window while reserving space for the answer, and then compares the cost of a long repeated prefix with and without prompt caching (using an example price and an example 90% discount).
# A tiny context-window manager: decide which messages still fitdef count_tokens(text): return max(1, round(len(text.split()) / 0.75)) # rough rule: 1 token is about 0.75 words def fit_to_window(system, messages, window, reserve_for_answer): budget = window - reserve_for_answer - count_tokens(system) kept, used = [], 0 for role, text in reversed(messages): # walk from the newest message backwards cost = count_tokens(text) if used + cost > budget: break # older messages no longer fit kept.append((role, text)) used += cost kept.reverse() return kept, used system = "You are a helpful tutor."messages = [ ("user", "Explain gradient descent " + "in detail " * 30), ("assistant", "Gradient descent is a method " + "that repeats small steps " * 25), ("user", "Now explain the learning rate " + "with an example " * 20), ("assistant", "The learning rate controls " + "how big each step is " * 20), ("user", "Summarize everything in one sentence."),]total = sum(count_tokens(m[1]) for m in messages)print("Whole conversation:", total, "tokens") for window in [2000, 400, 200]: kept, used = fit_to_window(system, messages, window, reserve_for_answer=50) print("Window", window, "-> kept", len(kept), "of", len(messages), "messages,", used, "tokens used") # Prompt caching: a repeated prefix can be billed at a discount (example discount: 90%)def cost_usd(prefix_tokens, new_tokens, price_per_million=3.0, cached=False): prefix_price = price_per_million * (0.1 if cached else 1.0) return (prefix_tokens * prefix_price + new_tokens * price_per_million) / 1_000_000 print("1,000 calls, 20,000-token prefix, no cache: USD", round(1000 * cost_usd(20000, 200), 2))print("1,000 calls, 20,000-token prefix, cached: USD", round(1000 * cost_usd(20000, 200, cached=True), 2))As the window shrinks from 2,000 to 400 to 200 tokens, fewer messages survive, and the oldest go first. The cost example shows why caching a large repeated prefix can cut the bill by roughly 90%.
Interactive Knowledge Check
Key Points
The Big Picture
Window = system prompt + history + documents + reserved answer
Treat every token as a limited resource. A clear, focused prompt is cheaper, faster and often more accurate than a giant one.
Up Next: Memory is the real limit of LLMs. In VRAM, Precision and Quantization you will learn how many bytes each number takes and how shrinking them lets big models fit on smaller GPUs.