Context Windows and the KV Cache

Learn what a context window really is, how to budget tokens, what to do when a conversation gets too long, why very long prompts cost more and can be less accurate, and how the KV cache and prompt caching affect memory and price.

25 minIntermediateCode Examples

The Core Idea: The context window is the model's shared working memory: prompt, history and answer all have to fit. Managing that budget well, and understanding the KV cache behind it, keeps conversations fast, cheap and accurate.

  1. Beginner: What Is a Context Window?

The context window is the maximum amount of text, measured in tokens, that a model can look at in one go. It is the model's working memory. Everything has to fit inside it: your system instructions, the whole conversation so far, any documents you paste in, and the answer the model is about to write.

A token is a piece of a word. A rough rule for English is that 1 token is about 0.75 words, so a 100,000-token window is roughly 75,000 words, about the length of a short novel.

Key point: The window is shared. If your prompt uses 7,000 tokens of an 8,000-token window, the answer can be at most about 1,000 tokens long.

  1. Beginner: What Happens When the Text Is Too Long?

A model has no memory between requests. A chat application re-sends the whole conversation with every new message. When that history no longer fits, something must be dropped. Applications use a few strategies:

StrategyHow it worksDownside
Truncate the oldestDrop the earliest messages firstForgets early instructions or facts
SummarizeReplace old messages with a short summarySummaries can lose details
Retrieve (RAG)Store everything outside, fetch only the relevant piecesNeeds a retrieval system
Pin important partsAlways keep the system prompt and key factsUses part of the budget permanently

Tip: Always keep the system prompt and the newest message. Drop or compress the middle of the history first.

  1. Medium: Longer Is Not Always Better

A bigger window is useful, but it is not free and it is not perfect:

CostPay per token
You pay for every input token on every request. Re-sending 50,000 tokens of history again and again adds up quickly.
SpeedSlower start
Reading a very long prompt takes longer before the first word appears.
AccuracyAttention is uneven
Research has found that models often use information at the start and end of a long prompt better than the middle.

Practical rule: Put the most important instructions at the beginning or end, and send only the context you need. A short, focused prompt often beats a huge one.

  1. Medium: The KV Cache and Prompt Caching

As you saw in the attention lesson, a model stores the keys and values of earlier tokens in a KV cache so it does not recompute them. That cache lives in GPU memory and grows with every token of every active conversation.

cache memory=2×layers×KV heads×head size×tokens×bytes\text{cache memory} = 2 \times \text{layers} \times \text{KV heads} \times \text{head size} \times \text{tokens} \times \text{bytes}

Prompt caching is a related service feature. If many requests start with the same long prefix (a big system prompt, a document, a set of examples), the provider can reuse the already-computed cache for that prefix. Cached input tokens are usually billed at a discount and are faster.

Design tip: Put the stable part of your prompt first and the changing part last. Then repeated requests share the same prefix and can hit the cache.

  1. Advanced: Managing Memory in Long Conversations

Serving many users with long contexts is mainly a memory problem. Common techniques:

TechniqueIdea
KV cache quantizationStore the cache in 8-bit or 4-bit numbers instead of 16-bit
Grouped-query attentionShare keys and values between heads (smaller cache by design)
Paged KV cacheStore the cache in small pages instead of one big block, to waste less memory
Prefix cachingReuse the cache of a shared beginning across requests
Sliding window / evictionKeep only recent tokens, or drop the least useful ones
Advertised vs. effective context
A model may accept 128,000 tokens but still answer worse near its limit. Test your own task at the real length you plan to use.

  1. Visual Explanation: The Context Budget

The figure shows who shares the context window and what to do when the conversation no longer fits.

Part 1 · Where the Window GoesEverything shares one budget
Input sideYour prompt
System prompt and rules
↓
Conversation history
↓
Pasted documents or retrieved chunks
↓
Prompt tokens
Output sideThe reply
Reserved space for the answer
↓
Model writes one token at a time
↓
Answer tokens (max_tokens)
Part 2 · When the History Is Too LongThree ways out
Option 1 · Truncate
Drop oldest
Fast and simple, but forgets.
Option 2 · Summarize
Compress
Keeps the gist in fewer tokens.
Option 3 · Retrieve
Fetch on demand
Only relevant pieces enter the window.
Click to see the budget formulaCompare ▼
Budget for history = window - reserved answer - system prompt. Walk backwards from the newest message and keep messages until the budget is full. The code below does exactly this.

Part 1 shows the two sides that compete for the same tokens. Part 2 compares the three common ways to make a long history fit. Open the bar for the budget rule used in the code.

  1. Python Implementation: A Context-Window Manager

This script estimates token counts, keeps the newest messages that fit in a given window while reserving space for the answer, and then compares the cost of a long repeated prefix with and without prompt caching (using an example price and an example 90% discount).

context_manager.pyPython 3.10+ · standard library only
# A tiny context-window manager: decide which messages still fitdef count_tokens(text):    return max(1, round(len(text.split()) / 0.75))       # rough rule: 1 token is about 0.75 words def fit_to_window(system, messages, window, reserve_for_answer):    budget = window - reserve_for_answer - count_tokens(system)    kept, used = [], 0    for role, text in reversed(messages):                # walk from the newest message backwards        cost = count_tokens(text)        if used + cost > budget:            break                                        # older messages no longer fit        kept.append((role, text))        used += cost    kept.reverse()    return kept, used system = "You are a helpful tutor."messages = [    ("user", "Explain gradient descent " + "in detail " * 30),    ("assistant", "Gradient descent is a method " + "that repeats small steps " * 25),    ("user", "Now explain the learning rate " + "with an example " * 20),    ("assistant", "The learning rate controls " + "how big each step is " * 20),    ("user", "Summarize everything in one sentence."),]total = sum(count_tokens(m[1]) for m in messages)print("Whole conversation:", total, "tokens") for window in [2000, 400, 200]:    kept, used = fit_to_window(system, messages, window, reserve_for_answer=50)    print("Window", window, "-> kept", len(kept), "of", len(messages), "messages,", used, "tokens used") # Prompt caching: a repeated prefix can be billed at a discount (example discount: 90%)def cost_usd(prefix_tokens, new_tokens, price_per_million=3.0, cached=False):    prefix_price = price_per_million * (0.1 if cached else 1.0)    return (prefix_tokens * prefix_price + new_tokens * price_per_million) / 1_000_000 print("1,000 calls, 20,000-token prefix, no cache: USD", round(1000 * cost_usd(20000, 200), 2))print("1,000 calls, 20,000-token prefix, cached:   USD", round(1000 * cost_usd(20000, 200, cached=True), 2))
EXPECTED OUTPUT (your numbers may differ slightly)
Whole conversation: 457 tokens
Window 2000 -> kept 5 of 5 messages, 457 tokens used
Window 400 -> kept 3 of 5 messages, 233 tokens used
Window 200 -> kept 1 of 5 messages, 7 tokens used
1,000 calls, 20,000-token prefix, no cache: USD 60.6
1,000 calls, 20,000-token prefix, cached: USD 6.6

As the window shrinks from 2,000 to 400 to 200 tokens, fewer messages survive, and the oldest go first. The cost example shows why caching a large repeated prefix can cut the bill by roughly 90%.

Interactive Knowledge Check

Question 1: A model has an 8,000-token window and your prompt is 7,500 tokens. About how long can the answer be?
Question 2: How should you arrange a prompt to benefit from prompt caching?

Key Points

✓The context window is measured in tokens and is shared by the prompt and the answer.
✓Chat apps re-send the whole history each turn, so long chats cost more and eventually overflow.
✓Fit long histories by truncating, summarizing or retrieving only what is relevant.
✓Longer is not always better: cost, speed and accuracy near the middle of long prompts can suffer.
✓The KV cache grows with every token. Prompt caching reuses a shared prefix at a discount.
✓Put stable content first and changing content last to benefit from caching.

The Big Picture

Context Budgeting in One Line

Window = system prompt + history + documents + reserved answer

Treat every token as a limited resource. A clear, focused prompt is cheaper, faster and often more accurate than a giant one.

Up Next: Memory is the real limit of LLMs. In VRAM, Precision and Quantization you will learn how many bytes each number takes and how shrinking them lets big models fit on smaller GPUs.