The Core Idea: A modern LLM is just one decoder block repeated many times. Compared with the 2017 Transformer, each part got a practical upgrade: RMSNorm before sublayers, a SwiGLU feed-forward, RoPE for position, GQA for a smaller cache, and sometimes Mixture of Experts to run only part of the model.
- Beginner: What Is a Modern LLM Made Of?
The original Transformer from 2017 had two halves: an encoder that reads and a decoder that writes. Today's chatbots such as GPT-style, Llama-style and Mistral-style models keep only the decoder half and stack it many times. This design is called a decoder-only Transformer.
A decoder-only model is just one block repeated, for example 32 times for a 7B model or 80 times for a 70B model. Each block has two jobs: an attention part that mixes information between tokens, and a feed-forward part that processes each token on its own.
Good news: Once you understand one block, you understand the whole model. The rest of this lesson is about what changed inside that block since 2017.
- Beginner: What Changed Since 2017
The overall shape stayed the same, but almost every small part was upgraded. Most modern open LLMs share these upgrades:
| Part | Original Transformer (2017) | Typical modern LLM | Why it changed |
|---|---|---|---|
| Overall design | Encoder + decoder | Decoder-only | Simpler, and next-token training scales very well |
| Normalization | LayerNorm after each sublayer | RMSNorm before each sublayer (pre-norm) | Cheaper and trains more stably in deep stacks |
| Feed-forward activation | ReLU | SwiGLU | Gating gives better quality for the same size |
| Position | Sinusoidal, added once | RoPE inside attention | Relative positions and longer context |
| Attention heads | Multi-head (MHA) | Grouped-query (GQA) | Smaller KV cache, faster generation |
| Bias terms | Used everywhere | Mostly removed | Fewer parameters, no loss in quality |
Remember: None of these changes is a new magic idea. Each one is a practical improvement that makes models cheaper to train, more stable, or faster to run.
- Medium: RMSNorm and SwiGLU, the Two Quiet Upgrades
Normalization keeps the numbers flowing through a deep network at a healthy size. LayerNorm subtracts the average and divides by the spread. RMSNorm skips the subtraction and only rescales by the root-mean-square (RMS), which is simpler and works just as well in practice.
The feed-forward part of each block used to be two matrices with a ReLU in the middle. SwiGLU uses three matrices: a gate decides how much of an up signal is allowed through, then a down matrix returns to the model size.
Plain English: Think of a water pipe with a tap. The up path is the water, the gate is the tap that decides how much flows, and down collects it into the right size pipe. Because the tap is learned, the network can switch information on and off per token.
- Medium: Pre-Norm Blocks and Residual Shortcuts
Each decoder block follows the same recipe. The input is normalized first, passed through a sublayer, and then added back to the original input. This add-back is the residual connection, a shortcut that lets information and gradients flow straight through the whole stack.
Why pre-norm? Putting the normalization before the sublayer keeps the shortcut path clean. That makes it much easier to train networks with dozens of layers without the signal blowing up or fading away.
- Advanced: Mixture of Experts (MoE)
Making a model bigger usually makes it slower, because every token passes through every parameter. A Mixture of Experts model breaks the feed-forward part into several smaller experts and adds a tiny router. For each token the router picks only a few experts, such as 2 out of 8, and the rest stay idle.
Trade-off: MoE models are cheap to run per token but still need memory for all experts, and they are trickier to train because the router must spread work evenly. Several well-known recent open and closed models use this design.
- Visual Explanation: One Modern Decoder Block
This is one decoder block, from input to output. Click each part to see what changed compared with the 2017 Transformer.
- Python Implementation: RMSNorm, SwiGLU and a Parameter Counter
This script builds the two quiet upgrades from scratch, RMSNorm and a SwiGLU feed-forward layer, and then counts the parameters of a Llama-style 7B model. The counter reproduces the well-known figure of about 6.74 billion and shows how GQA makes the model smaller.
import numpy as np rng = np.random.default_rng(0) def layer_norm(x, eps=1e-5): return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps) def rms_norm(x, weight, eps=1e-5): rms = np.sqrt(np.mean(x ** 2, axis=-1, keepdims=True) + eps) return x / rms * weight def silu(x): return x / (1 + np.exp(-x)) def swiglu_ffn(x, W_gate, W_up, W_down): return (silu(x @ W_gate) * (x @ W_up)) @ W_down # gate decides how much of "up" passes d_model, d_ff = 8, 16x = rng.normal(size=(3, d_model)) * 5 + 2 # 3 tokens with large, off-center valuesweight = np.ones(d_model) normed = rms_norm(x, weight)print("RMS of each token after RMSNorm:", np.round(np.sqrt(np.mean(normed ** 2, axis=-1)), 3))print("Largest mean left after LayerNorm:", round(float(np.abs(layer_norm(x).mean(-1)).max()), 6), "(LayerNorm also re-centers to zero)") W_gate = rng.normal(size=(d_model, d_ff)) * 0.3W_up = rng.normal(size=(d_model, d_ff)) * 0.3W_down = rng.normal(size=(d_ff, d_model)) * 0.3out = swiglu_ffn(normed, W_gate, W_up, W_down)print("SwiGLU output shape:", out.shape) # Count the parameters of a Llama-style decoderdef count_params(layers, d_model, d_ff, vocab, heads, kv_heads, tied_embeddings=False): head_dim = d_model // heads attention = 2 * d_model * d_model + 2 * d_model * kv_heads * head_dim # Q and O + K and V ffn = 3 * d_model * d_ff # gate, up, down norms = 2 * d_model embeddings = vocab * d_model output_head = 0 if tied_embeddings else embeddings return layers * (attention + ffn + norms) + embeddings + output_head + d_model mha = count_params(32, 4096, 11008, 32000, heads=32, kv_heads=32)gqa = count_params(32, 4096, 11008, 32000, heads=32, kv_heads=8)print("7B-style with multi-head attention:", format(mha, ","), "parameters")print("Same model with GQA (8 KV heads): ", format(gqa, ","), "parameters")The parameter count for the multi-head version is exactly 6,738,415,616, which matches the published size of a 7B Llama-style model. Switching to 8 KV heads removes about 0.8 billion parameters and, as you learned in the attention lesson, also shrinks the KV cache by 4×.
Interactive Knowledge Check
Key Points
The Big Picture
Embedding → N × (RMSNorm → GQA + RoPE → RMSNorm → SwiGLU) → Output head
You can now read the architecture description of almost any open LLM. Differences between models are mostly the number of layers, the width, the number of KV heads, and whether the feed-forward part is dense or Mixture of Experts.
Up Next: You know how an LLM is built. In LLM Pre-Training you will see how that empty network learns language from trillions of tokens, and how much compute that costs.