The Core Idea: A modern decoder-only LLM is built by stacking Transformer blocks. Each block uses causal self-attention to mix information across token positions and an MLP to transform the resulting representations. Residual connections and normalization help make the deep network trainable and stable.
1. Beginner: What Is a Transformer?
A Transformer is a neural-network architecture built around attention. It allows token representations to interact with one another and has become the foundation of most modern large language models.
For a decoder-only LLM, the high-level pipeline is:
INPUT TEXT
"The cat sat on the mat"
↓
TOKENIZATION
Text → token IDs
↓
TOKEN EMBEDDINGS + POSITIONAL INFORMATION
↓
TRANSFORMER BLOCKS
Attention + MLP + residual connections + normalization
↓
FINAL HIDDEN REPRESENTATIONS
↓
LANGUAGE MODEL HEAD
Hidden state → vocabulary logits
The Transformer does not directly store a sentence as a fixed symbolic representation. Instead, it repeatedly transforms vectors so that each position contains increasingly useful contextual information.
2. Beginner: Understanding Transformer Model Families
The original Transformer architecture introduced both an encoder and a decoder. Different model families use these components differently.
Encoder-only
Primarily designed to build contextual representations of an input sequence.
Example: BERT
Encoder-decoder
Uses one network to encode an input and another to generate an output sequence.
Example: T5
Decoder-only
Generates tokens autoregressively while attending to the available context.
Example: GPT-style LLMs
For modern text-generation LLMs, the architecture you should understand first is the decoder-only Transformer. It uses causal self-attention so a position cannot use information from future tokens.
Important: "Decoder-only" does not mean the model is incomplete. It means the model uses the decoder-style stack without a separate encoder stack.
3. Beginner: What Is a Transformer Block?
A Transformer is not one single attention operation. It is usually a stack of repeated Transformer blocks.
A simplified decoder-only block contains:
1. Normalization
Often pre-normalization in modern LLMs.
2. Causal Self-Attention
Mixes information across allowed token positions.
3. Residual Connection
Adds the attention update back to the stream.
4. MLP / FFN
Transforms each contextualized token representation.
5. Another Residual Connection
Combines the MLP output with the existing stream.
6. Normalization
The exact placement depends on the architecture.
INPUT REPRESENTATIONS
↓
NORMALIZATION
↓
CAUSAL SELF-ATTENTION
↓
RESIDUAL ADDITION
↓
MLP / FEED-FORWARD NETWORK
↓
RESIDUAL ADDITION
↓
OUTPUT REPRESENTATIONS
The exact ordering varies across model families. Modern LLMs commonly use pre-normalization and may use RMSNorm and gated MLPs such as SwiGLU.
4. Beginner: Causal Self-Attention
Self-attention allows token representations to use information from other positions in the sequence. In an autoregressive LLM, the attention must be causal.
For the sequence:
The cat sat on the mat
The
×
×
×
×
×
The
cat
×
×
×
×
The
cat
sat
×
×
×
Each position can use itself and earlier positions, but not future positions.
This restriction is implemented with a causal attention mask. Without it, a training example could allow the model to see the token it is supposed to predict, which would break the autoregressive learning objective.
5. Intermediate: Query, Key, and Value
Self-attention creates three learned projections from the input representations: Query, Key, and Value.
Query
Represents what information the current position is looking for.
Key
Represents what each position offers for matching against a query.
Value
Contains the information that is actually combined when attention is applied.
A simplified attention calculation is:
Attention = softmax of (Q × K-transpose divided by square-root of d-k) × V
The query-key interaction determines how strongly positions relate to one another. The resulting weights are then used to combine the value vectors.
Why scale by square-root of d-k? The scaling helps prevent attention scores from becoming excessively large as the key/query dimensionality increases.
6. Intermediate: Multi-Head Attention
Transformers normally use multiple attention heads instead of one large attention operation. Each head works in a learned subspace and can capture different relationships.
INPUT REPRESENTATIONS
↓
HEAD 1
Attention pattern
HEAD 2
Attention pattern
HEAD N
Attention pattern
↓ concatenate
OUTPUT PROJECTION
↓
ATTENTION OUTPUT
Different heads can learn to emphasize different relationships, such as local dependencies, long-range relationships, syntactic patterns, or references between entities.
Modern LLMs can also use attention variants such as Multi-Query Attention and Grouped-Query Attention. These reduce key/value memory requirements during inference while retaining multiple query heads.
7. Intermediate: Feed-Forward Network / MLP
Attention mixes information across token positions. The feed-forward network, commonly called the MLP, then transforms the representation at each position.
Simplified MLP
INPUT
→
EXPANSION + ACTIVATION
→
PROJECTION
A simplified form is:
MLP of x = W2 × activation of (W1 × x + b1) + b2
The MLP is applied independently at each position, but the input representation already contains contextual information produced by attention.
Modern LLMs often use gated MLP variants such as SwiGLU instead of a simple activation function. The exact dimensions and activation design vary across model families.
8. Intermediate: Residual Connections
A residual connection adds a layer's output back to the representation that entered the layer.
INPUT x
↓
ATTENTION OR MLP: f of x
↓
ADD x + f of x
↓
UPDATED REPRESENTATION
In simplified mathematical notation, the idea is:
x-new = x + f of x
Residual pathways help information and gradients travel through very deep networks. Instead of forcing every layer to completely replace the existing representation, each sub-layer can learn an update to it.
9. Intermediate: Layer Normalization and RMSNorm
Normalization helps control the scale of activations and makes optimization more stable.
LayerNorm
Normalizes activations using statistics based on the mean and variance of the features.
RMSNorm
Uses the root-mean-square magnitude of the activations without explicitly centering by the mean.
Many modern decoder-only LLMs use RMSNorm. The exact normalization design is architecture-specific.
Pre-norm vs post-norm: The position of normalization relative to attention, MLP, and residual operations is an architectural choice. Modern LLMs commonly use a pre-norm style.
10. Intermediate: One Token's Representation Through a Block
Consider the token "cat" in the sequence:
The cat sat on the mat
Its initial representation comes from its token embedding plus positional information. Inside a Transformer block, that representation is repeatedly updated.
Step 1
Normalize the representation.
Step 2
Use causal self-attention to retrieve useful context.
Step 3
Add the attention update through the residual path.
Step 4
Normalize before the MLP in a pre-norm architecture.
Step 5
Transform the representation through the MLP.
Step 6
Add the MLP update through another residual path.
After many Transformer blocks, the representation of "cat" is no longer simply the original embedding. It contains information shaped by the surrounding context and by the computations performed throughout the network.
11. Intermediate: Stacking Transformer Blocks
Large language models stack many Transformer blocks. The output of one block becomes the input to the next.
TOKEN EMBEDDINGS
↓
TRANSFORMER BLOCK 1
↓
TRANSFORMER BLOCK 2
↓
TRANSFORMER BLOCK 3
↓
⋮
↓
TRANSFORMER BLOCK N
↓
FINAL NORMALIZATION
Depth gives the network repeated opportunities to transform the representation. The exact number of blocks differs substantially between model families and sizes.
12. Intermediate: The Language Model Head
After the final Transformer block, the model has a hidden representation for each token position. For next-token prediction, the representation at the relevant position is projected into the vocabulary space.
FINAL HIDDEN STATE
↓
LINEAR PROJECTION
↓
VOCABULARY LOGITS
one score for every vocabulary token
↓
SOFTMAX / DECODING
↓
NEXT TOKEN
A simplified projection can be written as:
logits = hidden-state × output-weights-transpose + bias
The resulting logits are then passed to the decoding process you learned in the previous module: probabilities are formed and a token is selected using greedy decoding or a sampling strategy.
Connection to Module 04: The Transformer architecture computes the hidden representations and logits. The generation algorithm then turns those logits into the next token.
13. Visual Explanation: Complete LLM Architecture
The complete decoder-only architecture can be viewed as a sequence of transformations from text to contextual representations and finally to vocabulary logits.
INPUT TEXT
↓
TOKENIZATION
↓
TOKEN EMBEDDINGS + POSITIONAL INFORMATION
↓
TRANSFORMER BLOCK
Normalization
↓
Causal Self-Attention
↓
Residual Connection
↓
MLP / Feed-Forward Network
↓
Residual Connection
↓ repeat many times
TRANSFORMER BLOCK N
↓
FINAL NORMALIZATION
↓
LANGUAGE MODEL HEAD
↓
VOCABULARY LOGITS
↓
NEXT TOKEN
Attention
Moves information between token positions.
MLP
Transforms each contextualized token representation.
LM Head
Maps hidden representations to vocabulary scores.
One important distinction is that the Transformer blocks produce contextual representations, while the language model head converts those representations into scores used for next-token prediction.
14. Code: Inspect a Transformer Architecture with Python
The Hugging Face Transformers library lets you inspect the configuration and components of a pretrained causal language model. The example below uses GPT-2 because its architecture is small enough to inspect while still demonstrating the same core Transformer ideas.
Install the library
pip install transformers torchExample: Inspect Model Configuration
from transformers import AutoConfig, AutoModelForCausalLM model_name = "gpt2"config = AutoConfig.from_pretrained(model_name) print("Hidden size:", config.n_embd)print("Number of layers:", config.n_layer)print("Attention heads:", config.n_head)print("Vocabulary size:", config.vocab_size) model = AutoModelForCausalLM.from_pretrained(model_name)print("Model loaded:", type(model).name)Output
Hidden size: 768Number of layers: 12Attention heads: 12Vocabulary size: 50257Model loaded: GPT2LMHeadModelWhat to notice: The configuration exposes important architectural dimensions. Hidden size describes the width of token representations, the layer count describes how many Transformer blocks are stacked, attention heads describe the parallel attention subspaces, and vocabulary size determines how many token scores the language model head produces.
Example: Inspect the Model Structure
print(model.transformer)What you should look for: GPT-2 exposes a stack of Transformer blocks, attention modules, feed-forward modules, normalization components, token embeddings, positional embeddings, and a final language-model projection.
The exact class names and internal attributes vary across model families. Do not assume that every modern LLM uses the same module names or the same normalization and MLP implementation.
15. Intermediate: Putting the Architecture Together
It helps to separate the Transformer into three conceptual responsibilities.
Context Mixing
Causal self-attention determines which earlier positions can contribute information to each current position.
Representation Transformation
The MLP transforms the contextualized representation at each position.
Prediction
The final hidden state is projected into vocabulary logits for next-token prediction.
TOKENS
↓
EMBEDDINGS
↓
ATTENTION: MIX CONTEXT
↓
MLP: TRANSFORM REPRESENTATIONS
↓ repeat across blocks
FINAL HIDDEN STATE
↓
VOCABULARY LOGITS
This architecture is the computational foundation underneath the generation loop from the previous lesson. Generation repeatedly feeds the growing context through this stack and uses the resulting logits to select the next token.
16. Key Architecture Terms to Remember
| Term | Meaning |
|---|---|
| Transformer | Neural architecture centered around attention and repeated representation transformations. |
| Causal Self-Attention | Self-attention restricted so each position cannot use future tokens. |
| Query | Represents what information a position is looking for. |
| Key | Represents what information a position offers for matching. |
| Value | The information combined according to attention weights. |
| Attention Head | One learned attention subspace within multi-head attention. |
| MLP / FFN | Position-wise network that transforms contextualized token representations. |
| Residual Connection | Adds a sub-layer update back to the existing representation. |
| LayerNorm / RMSNorm | Normalization methods used to stabilize Transformer computations. |
| Transformer Block | Repeated unit containing attention, MLP, residual paths, and normalization. |
| Language Model Head | Projection from hidden representations into vocabulary logits. |
The complete picture: Tokenization produces token IDs. Embeddings create vectors. Transformer blocks repeatedly mix context and transform representations. The final hidden states are projected into vocabulary logits. The generation process then selects the next token and repeats.
⚡ Knowledge Check
Which component allows a token representation to incorporate information from other token positions?
A) MLP only▼
✕ Incorrect. The MLP transforms each position independently.
B) Self-attention▼
✓ Correct! Self-attention allows information to be mixed across the permitted token positions.
Module Complete
You now understand the core architecture behind decoder-only LLMs.
You should now be comfortable with decoder-only Transformers, Transformer blocks, causal self-attention, Query-Key-Value attention, multi-head attention, MLPs, residual connections, normalization, stacked blocks, and the language model head.