The Core Idea: An LLM only produces probabilities for the next token. A sampling method chooses one. Temperature, top-k and top-p let you trade predictability for variety without retraining anything.
- Beginner: The Model Only Gives Probabilities
An LLM does not produce a sentence directly. At every step it produces a score for every word in its vocabulary, which is turned into a probability distribution. Something then has to pick one word. That choice rule is called a decoding strategy or sampling method.
The same model can sound careful, boring, creative or chaotic depending only on this rule. No retraining is needed.
- Beginner: Greedy Decoding and Why It Is Not Enough
The simplest rule is greedy decoding: always take the most likely token. It is fast and gives the same answer every time, which is great for facts and code.
But greedy has two problems. It can get stuck in repetitive loops, and it makes writing feel flat because it never takes a less obvious but still good word. Sampling fixes this by choosing randomly according to the probabilities, so likely words win most of the time but not always.
Beam search: Another classic method keeps the best few partial sentences at once. It works well for translation, but for open-ended chat it tends to produce dull text, so chat models usually sample instead.
- Medium: Temperature, the Creativity Dial
Temperature divides the scores before the softmax. It reshapes the probabilities without changing their order:
| Temperature | Effect | Good for |
|---|---|---|
| Close to 0 | The top token gets almost all the probability. Behaves like greedy. | Facts, extraction, code, math |
| Around 0.7 | Mostly the top choices, with some variety | General chat and writing |
| 1.0 | The model's natural distribution | Brainstorming |
| Above 1.5 | Rare tokens become likely. Text may turn into nonsense. | Rarely useful |
Remember: Low temperature does not make the model smarter. It only makes it more predictable.
- Medium: Top-k and Top-p, Cutting Off the Bad Tail
Even at a sensible temperature, a vocabulary of 100,000 tokens has a long tail of silly words. Each one has a tiny probability, but together they can add up and occasionally derail a sentence. Two filters remove the tail:
Practical default: Many applications use temperature around 0.7 with top-p around 0.9 to 0.95. Change one knob at a time so you can see what it does.
- Advanced: Repetition, Seeds and Stopping
Real systems add a few more controls around the main sampler:
| Control | What it does |
|---|---|
| Repetition / frequency penalty | Lowers the score of tokens already used, to avoid loops |
| Seed | Fixes the random choices so a sampled answer can be repeated (not always guaranteed across hardware) |
| Stop sequences | Ends generation when a chosen string appears, such as a closing tag |
| Max tokens | A hard limit on the length of the answer |
| Min-p and similar | Newer filters that keep tokens above a fraction of the top probability |
- Visual Explanation: How a Word Gets Chosen
The figure follows one next-word decision using the numbers from the code below: five candidate words with scores that become probabilities.
Click to see a quick rule of thumbCompare ▼
In Part 1, every card starts from the same probabilities and applies a different rule. Part 2 summarizes what each dial changes. Open the bar for a rule of thumb.
- Python Implementation: Temperature, Top-k and Top-p
This script builds each sampling method from scratch on a five-word example, then draws 1,000 samples to show how the choices spread out.
import numpy as np words = ["the", "a", "cat", "dog", "banana"]logits = np.array([3.0, 2.5, 2.0, 1.0, -1.0]) # raw scores from a model def softmax(z, temperature=1.0): z = z / temperature z = z - z.max() e = np.exp(z) return e / e.sum() def show(title, p): print(title.ljust(22), {w: round(float(x), 3) for w, x in zip(words, p)}) # 1. Temperature reshapes the probabilitiesfor T in [0.5, 1.0, 2.0]: show("temperature " + str(T), softmax(logits, T)) # 2. Top-k: keep only the k most likely words, then renormalizedef top_k(p, k): keep = np.argsort(p)[::-1][:k] out = np.zeros_like(p) out[keep] = p[keep] return out / out.sum() show("top-k (k=2)", top_k(softmax(logits), 2)) # 3. Top-p (nucleus): keep the smallest set whose probabilities add up to pdef top_p(p, threshold): order = np.argsort(p)[::-1] cumulative = np.cumsum(p[order]) cutoff = np.searchsorted(cumulative, threshold) + 1 out = np.zeros_like(p) out[order[:cutoff]] = p[order[:cutoff]] return out / out.sum() show("top-p (p=0.9)", top_p(softmax(logits), 0.9)) # 4. Greedy vs sampling over 1,000 drawsrng = np.random.default_rng(0)p = softmax(logits)draws = rng.choice(words, size=1000, p=p)print("Greedy always picks:", words[int(np.argmax(p))])print("Sampling 1,000 times:", {w: int((draws == w).sum()) for w in words})Notice how a low temperature piles probability on the, while a high temperature spreads it out. Top-k with keeps just two words, while top-p keeps three because it needs three to reach 0.9. In the 1,000 draws, the wins about 44% of the time but not always.
Interactive Knowledge Check
Key Points
The Big Picture
Scores → probabilities → (temperature, top-k, top-p) → pick one token → repeat
Two users can run the same model and get very different feelings simply by changing these dials. Knowing them lets you debug strange or repetitive output quickly.
Up Next: Every token you generate, and every token in your prompt, has to fit inside a limit. In Context Windows and the KV Cache you will learn how to budget that space and keep long conversations affordable.