The Core Idea: Alignment teaches a model what people prefer. People compare two answers, a model learns from those comparisons, and the model is nudged toward the winners, either with a reward model and RLHF or directly with DPO.
- Beginner: Why a Smart Model Is Not Yet a Good Assistant
After pre-training and fine-tuning, an LLM can write fluent text. But fluent is not the same as helpful, honest and safe. A model may give a confident wrong answer, follow a harmful request, ramble, or flatter the user. Alignment is the work of steering a model toward what people actually want.
The hard part is that good is difficult to write down as a rule. It is much easier for a person to look at two answers and say which one is better. Almost all modern alignment methods are built on that simple idea: learning from preferences.
Analogy: Pre-training builds a very knowledgeable student. Alignment is the mentoring that teaches the student how to behave in front of real people.
- Beginner: Collecting Human Preferences
The raw material for alignment is a preference dataset. For each prompt, the model writes two or more answers, and a person (or sometimes another AI) marks which one is better:
| Prompt | Answer A | Answer B | Better |
|---|---|---|---|
| How do I boil an egg? | Put it in boiling water for 8 to 10 minutes, then cool it in cold water. | Just google it. | A (chosen) |
| Is the Earth flat? | No. Satellite photos, ship horizons and flights all show it is round. | Some people say yes! | A (chosen) |
Each pair gives a chosen answer and a rejected answer. Thousands to hundreds of thousands of such pairs are used. The quality and diversity of the labelers matters a lot, because the model will learn their idea of good.
AI feedback: Collecting human labels is slow and costly, so some methods let a strong model judge the answers using a written list of principles (for example Constitutional AI). This is often called RLAIF.
- Medium: RLHF, Teaching a Reward Model
RLHF (Reinforcement Learning from Human Feedback) was the first widely used recipe. It has three stages:
The reward model learns from pairs using the Bradley-Terry idea: the probability that answer A beats answer B depends on the difference of their scores.
The leash: If the language model could chase reward freely it would find tricks that fool the reward model (called reward hacking). So training adds a KL penalty that keeps the new model close to the original one.
- Medium: DPO, Skipping the Reward Model
RLHF works, but it is complex: three models, an unstable reinforcement-learning loop and many settings to tune. DPO (Direct Preference Optimization) reaches a similar goal with one simple loss that is trained like normal fine-tuning:
In plain words: compare the model with a frozen reference copy. Raise the probability of the chosen answer relative to the reference, lower the probability of the rejected answer , and the knob controls how far the model may drift.
| RLHF (PPO) | DPO | |
|---|---|---|
| Models needed | Policy, reference, reward model (and a value model) | Policy and a frozen reference |
| Training | Reinforcement-learning loop, sampling needed | Plain supervised-style training on pairs |
| Complexity | High and sometimes unstable | Much simpler |
| Used when | Maximum control, large teams | Quick, stable preference tuning |
- Advanced: Failure Modes and Safety Limits
Alignment is not a finished problem. Models trained on human approval can pick up unwanted habits:
| Problem | What it looks like | Common mitigation |
|---|---|---|
| Reward hacking | Longer or more confident answers that score well but are not better | KL penalty, better reward models, human spot checks |
| Sycophancy | Agreeing with the user even when the user is wrong | Preference data that rewards polite disagreement |
| Over-refusal | Refusing harmless requests | Balanced data with both safe and helpful examples |
| Preference noise | Labelers disagree with each other | Clear guidelines, several labelers, quality audits |
| Jailbreaks | Prompts that trick the model into ignoring its training | Red-teaming and extra safety layers |
Remember: Alignment shapes behavior but does not make a model perfect. Real systems combine aligned models with testing, monitoring and extra safeguards.
- Visual Explanation: Two Roads to an Aligned Model
Both roads start from the same preference pairs. The figure compares the steps each one needs, then shows where the data comes from.
Click to see how the DPO loss behavesCompare ▼
Read Part 1 left to right: RLHF has one extra learned model (the reward model), DPO does not. Part 2 shows the shared starting point: pairs of answers with a chosen winner. Open the bar at the bottom to see how the DPO loss moves.
- Python Implementation: A Reward Model and the DPO Loss
This script does two things from scratch. First it trains a tiny reward model from noisy human choices between pairs of answers. Then it computes the DPO loss for four different situations.
import numpy as np def sigmoid(x): return 1 / (1 + np.exp(-x)) rng = np.random.default_rng(0) # 1. Reward model: learn what humans prefer from pairs of answers (Bradley-Terry)# Each answer is described by 2 features: [helpfulness, rudeness]w_true = np.array([2.0, -3.0]) # humans like helpful, dislike rudeA = rng.normal(size=(500, 2)) # answer A featuresB = rng.normal(size=(500, 2)) # answer B featuresprefer_A = (sigmoid((A - B) @ w_true) > rng.random(500)).astype(float) # noisy human choice w = np.zeros(2)for step in range(500): p = sigmoid((A - B) @ w) # model's chance that A beats B grad = (A - B).T @ (p - prefer_A) / len(p) # gradient of the logistic loss w -= 0.5 * gradprint("Learned reward weights [helpful, rude]:", np.round(w, 2), "| humans secretly used", w_true)agree = np.mean((sigmoid((A - B) @ w) > 0.5) == (prefer_A == 1))print("Reward model agrees with human choices:", round(float(agree), 3)) # 2. DPO loss: no reward model needed, only log-probabilities of the two answersdef dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta=0.1): margin = beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected)) return -np.log(sigmoid(margin)) ref_c, ref_r = -12.0, -12.0 # reference (frozen SFT model) log-probsfor name, pc, pr in [("unchanged policy", -12.0, -12.0), ("likes chosen more", -10.0, -12.0), ("likes chosen a lot", -6.0, -18.0), ("prefers the REJECTED", -14.0, -9.0)]: print(name, "-> DPO loss", round(float(dpo_loss(pc, pr, ref_c, ref_r)), 3))The learned weights point the same way as the hidden human preferences (helpful is good, rude is bad), and the model agrees with most choices even though the humans were noisy. For DPO, the loss starts at 0.693 when nothing has changed, shrinks as the policy favors the chosen answer, and grows when it favors the rejected one.
Interactive Knowledge Check
Key Points
The Big Picture
Compare answers → learn what people prefer → nudge the model toward it (but not too far)
Pre-training gives knowledge, fine-tuning gives format, and alignment gives judgment about what a good answer is. All three together make a usable assistant.
Up Next: A trained, aligned model still has to choose each next word. In Text Generation and Sampling you will see how temperature, top-k and top-p turn probabilities into text.