How LLMs Generate Text

Understand how an LLM turns a prompt into probabilities, selects the next token, and repeats the process to generate text.

30 minBeginner

The Core Idea: An LLM does not generate an entire response in one operation. It predicts one next token at a time. The selected token is added to the context, the model predicts again, and the loop continues until generation stops.

1. Beginner: From Prompt to Prediction

When you send a prompt, the model follows the pipeline you have already learned: text → tokens → token IDs → embeddings → Transformer layers. The final representation is then converted into scores for the vocabulary.
USER PROMPT
"The capital of France is"
↓
TOKENIZATION
Text → token IDs
↓
EMBEDDINGS + POSITIONAL INFORMATION
↓
TRANSFORMER LAYERS
Attention + feed-forward processing
↓
LOGITS
A score for every vocabulary token
↓
NEXT-TOKEN SELECTION
Example: "Paris"
Input
The current sequence of tokens.
Prediction
Scores describing possible next tokens.
Selection
A decoding strategy chooses the next token.

2. Beginner: Autoregressive Generation

A decoder-only LLM is autoregressive: it uses the tokens already in the context to predict the next token.
One Token at a Time
"The capital of France"
↓ predict
"is"
↓ append
"The capital of France is"
↓ predict
"Paris"
The generated token becomes part of the context for the next step. This is why a long response requires many sequential generation steps.
⚡ Knowledge Check
What makes generation autoregressive?
A) Every token is independent▼
✕ Incorrect. Later predictions use the accumulated context.
B) Previous tokens affect later predictions▼
✓ Correct! Generated tokens become part of the next context.

3. Beginner: Logits

After the Transformer processes the current sequence, the model produces a vector of logits. There is one score for every token in the vocabulary.
Example Scores
Paris8.4
London5.1
Berlin4.7
banana-2.3
Logits are not probabilities. They can be any real-valued numbers. A softmax transformation converts them into a probability distribution.
Mental model: logits are raw preference scores before normalization.

4. Beginner: Softmax and Probabilities

Softmax transforms logits into a probability distribution. The probabilities are non-negative and sum to 1.
Logits → Softmax → Probabilities
Raw scores
[8.4, 5.1, 4.7, -2.3]
→ softmax →
Distribution
[high, lower, lower, tiny]
The decoding stage uses this distribution to decide which token becomes the next part of the response.

5. Intermediate: Temperature

Temperature controls the sharpness of the probability distribution used for sampling. Lower values make high-probability choices more dominant; higher values make lower-ranked alternatives more competitive.
Low
More concentrated and generally more predictable.
≈ 1
Uses the model distribution without strong sharpening or flattening.
High
Flatter distribution and more alternative choices.
Illustrative Distribution
Lower temperature
A0.82
B0.10
C0.05
Others0.03
Higher temperature
A0.48
B0.23
C0.16
Others0.13
Important: Temperature changes inference-time selection behavior. It does not retrain or modify the model weights.

6. Intermediate: Greedy Decoding

Greedy decoding selects the highest-probability token at every step.
Highest Probability Wins
Token A — 0.62
Token B — 0.21
Token C — 0.17
↓
Select Token A
Greedy decoding is simple and deterministic for a fixed input, but it never explores lower-probability alternatives.

7. Intermediate: Top-k Sampling

Top-k sampling keeps only the k highest-probability candidates and samples from that smaller set.
Example: k = 3
1. Token A — 0.42
2. Token B — 0.25
3. Token C — 0.17
Keep A, B, C → sample from them
Other candidates → excluded
Top-k controls the number of candidates. After filtering, the remaining distribution can be renormalized for sampling.

8. Intermediate: Top-p Sampling

Top-p, or nucleus sampling, keeps the smallest set of high-probability tokens whose cumulative probability reaches a chosen threshold p.
Example
A — 0.50
B — 0.27
C — 0.12
D — 0.06
E — 0.05
p = 0.89 → keep enough candidates to reach the threshold
Unlike top-k, the number of retained tokens can change at every generation step because the probability distribution changes.
Top-k
Fixed number of candidates.
Top-p
Variable number of candidates based on cumulative probability.

9. Intermediate: The Complete Generation Loop

The most important mental model is the repeated generation loop.
CURRENT CONTEXT
↓
TRANSFORMER FORWARD PASS
↓
LOGITS → PROBABILITIES
↓
DECODING
greedy / sampling / top-k / top-p
↓
NEXT TOKEN
↓
APPEND TOKEN → REPEAT
Generation stops when a stopping condition is reached, such as an end-of-sequence token, a maximum number of new tokens, or an application-specific stop sequence.

10. Code: Generate Text with Python

The example below uses Hugging Face Transformers with a pretrained causal language model. Sampling is enabled so you can see how decoding settings affect generation.
Install the library
pip install transformers torch
Example: Generate a Continuation
from transformers import AutoTokenizer, AutoModelForCausalLM model_name = "gpt2"tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModelForCausalLM.from_pretrained(model_name) prompt = "Artificial intelligence is"inputs = tokenizer(prompt, return_tensors="pt") output_ids = model.generate(    **inputs,    max_new_tokens=20,    do_sample=True,    temperature=0.7,    top_p=0.9,) generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)print(generated_text)
Output
Artificial intelligence is changing the way people work and learn.
What to notice: The exact continuation can vary because sampling is enabled. The model generates new tokens sequentially until the generation limit or another stopping condition is reached.
Example: Greedy Generation
output_ids = model.generate(    **inputs,    max_new_tokens=20,    do_sample=False,) print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Compare: do_sample=False uses greedy-style selection, while do_sample=True allows sampling and makes temperature/top-p relevant.

11. Visual Explanation: One Generation Step

This is the complete mental model for one generation step.
1. CURRENT TOKENS
↓
2. TRANSFORMER FORWARD PASS
↓
3. LOGITS
one score per vocabulary token
↓
4. PROBABILITIES
↓
5. DECODING
temperature / top-k / top-p / greedy
↓
6. NEXT TOKEN
↓
APPEND → REPEAT
Model
Computes next-token scores.
Decoder
Chooses a token from the distribution.
Loop
Appends the token and repeats.

12. Intermediate: When Does Generation Stop?

An LLM needs a stopping condition. Common mechanisms include an end-of-sequence token, a maximum number of new tokens, or an application-specific stop sequence.
End Token
The model generates a special termination token.
Max New Tokens
Generation stops after a configured number of new tokens.
Stop Sequence
A serving application can stop when a custom sequence appears.
Context limit: Generation must also respect the model's supported context capacity.

13. Intermediate: Why Outputs Can Vary

With sampling enabled, the model can select different tokens from the probability distribution. Therefore the same prompt can produce different continuations.
Sampling Introduces Choice
Candidate A
0.55
Candidate B
0.30
Candidate C
0.15
Greedy decoding always selects the highest-probability option. Sampling can select another candidate, which creates variation across runs.

14. Key Terms to Remember

TermMeaning
AutoregressiveUsing previous tokens to generate later tokens.
LogitsRaw scores produced for vocabulary tokens.
SoftmaxConverts logits into a probability distribution.
TemperatureControls distribution sharpness during sampling.
Greedy DecodingSelects the highest-probability token.
Top-kSamples only from the k highest-probability candidates.
Top-pSamples from the smallest candidate set reaching cumulative probability p.
SamplingSelects a token probabilistically from the allowed distribution.
The complete picture: Tokenization gives token IDs. Embeddings turn IDs into vectors. Transformers process the sequence. The model produces logits. Decoding turns the distribution into the next token. That token is appended, and the process repeats.
Module Complete
You now understand the basic LLM generation loop.
You should now be comfortable with autoregressive generation, logits, softmax, temperature, greedy decoding, top-k, top-p, sampling, stopping conditions, and the path from a prompt to generated tokens.