The Core Idea: When a language model writes a sentence, every new word is chosen by looking back at the words that came before it. Causal masking is the rule that stops the model from looking at future words. The KV cache is the memory trick that stops it from redoing old work at every step. Together they make training fair and generation fast.
1. Beginner: Attention as a Smart Lookup
Key question for this lesson: which tokens are the Keys and Values allowed to come from? The answer is the whole point of causal masking.
2. Beginner: The No-Peeking Rule
Analogy: reading a book with a sliding card that covers everything to the right of your current word. You may re-read anything on the left as often as you like, but the right side stays hidden.
3. Intermediate: The Math of Causal Masking
| Row reads → | Token 1 | Token 2 | Token 3 | Token 4 |
|---|---|---|---|---|
| Token 1 | ✓ | −∞ | −∞ | −∞ |
| Token 2 | ✓ | ✓ | −∞ | −∞ |
| Token 3 | ✓ | ✓ | ✓ | −∞ |
| Token 4 | ✓ | ✓ | ✓ | ✓ |
Worked example. Suppose token 3 has raw scores 2.0, 1.0, 0.5 against tokens 1, 2, 3, and −∞ against token 4 after masking. Softmax uses , , and . The sum is about 11.76, so the weights are roughly 0.629, 0.231, 0.140 and 0. They add up to 1, and the future token gets no say.
4. Intermediate: Why Generation Gets Slow
| Tokens generated | Positions processed without cache | Positions processed with cache |
|---|---|---|
| 10 | 55 | 10 |
| 100 | 5,050 | 100 |
| 1,000 | 500,500 | 1,000 |
The key observation. Because of the causal mask, the output of token depends only on tokens to . When a new token arrives at the end, the Keys and Values of all older tokens do not change at all, since the model weights are frozen during generation. Old work can be saved and reused. This only works because of causal masking: in a model where old tokens could read new ones, every old vector would change at each step and nothing could be reused.
5. Advanced: The KV Cache
| Context length | Cache size for ONE sequence |
|---|---|
| 1,024 tokens | 0.5 GiB |
| 4,096 tokens | 2 GiB |
| 32,768 tokens | 16 GiB |
6. Visual Explanation: Masking and Caching Side by Side
Click to see why only K and V are cachedExplain ▼
7. Python Implementation: Causal Attention with a KV Cache
import numpy as np np.random.seed(0) # Tiny sizes so the output stays readabled_model = 8d_k = 4seq_len = 5 # Random projection weights (frozen, like a trained model)W_q = np.random.randn(d_model, d_k)W_k = np.random.randn(d_model, d_k)W_v = np.random.randn(d_model, d_k)X = np.random.randn(seq_len, d_model) def softmax(scores): scores = scores - scores.max(axis=-1, keepdims=True) e = np.exp(scores) return e / e.sum(axis=-1, keepdims=True) # Training style: all tokens at once, with a causal maskdef full_causal_attention(X): n = X.shape[0] Q = X @ W_q K = X @ W_k V = X @ W_v scores = Q @ K.T / np.sqrt(d_k) mask = np.triu(np.ones((n, n)), k=1) == 1 scores = np.where(mask, -np.inf, scores) weights = softmax(scores) return weights @ V, weights # Generation style: a cache holding all past keys and valuesdef new_cache(): return {"K": np.zeros((0, d_k)), "V": np.zeros((0, d_k))} def decode_step(x_new, cache): q = x_new @ W_q k = x_new @ W_k v = x_new @ W_v cache["K"] = np.vstack([cache["K"], k]) cache["V"] = np.vstack([cache["V"], v]) scores = cache["K"] @ q / np.sqrt(d_k) weights = softmax(scores) return weights @ cache["V"] full_out, full_w = full_causal_attention(X) cache = new_cache()cached_out = np.array([decode_step(X[t], cache) for t in range(seq_len)]) print("Outputs match:", np.allclose(full_out, cached_out))print("Future weights are zero:", np.allclose(np.triu(full_w, k=1), 0))print("Rows sum to 1:", np.allclose(full_w.sum(axis=1), 1))print("First token attends only to itself:", np.round(full_w[0], 3))print("Cache shape after 5 tokens:", cache["K"].shape) # Memory cost of a real-sized cache (7B-class settings, 16-bit values)layers = 32kv_heads = 32head_dim = 128bytes_per_value = 2per_token = 2 * layers * kv_heads * head_dim * bytes_per_valueprint("KV cache per token (KiB):", per_token / 1024)print("KV cache for 4096 tokens (GiB):", per_token * 4096 / (1024 * 1024 * 1024))Outputs match: TrueFuture weights are zero: TrueRows sum to 1: TrueFirst token attends only to itself: [1. 0. 0. 0. 0.]Cache shape after 5 tokens: (5, 4)KV cache per token (KiB): 512.0KV cache for 4096 tokens (GiB): 2.0Knowledge Check
Key Points
The Big Picture
Up Next: Decoding Strategies: Temperature, Top-k and Top-p Sampling. Once the model has computed its output for the next position, how do we actually pick the token? That choice shapes how creative or predictable the text feels.