Alignment and Preference Optimization: RLHF and DPO

Learn how language models are steered toward helpful, honest and safe behavior: preference data, reward models, RLHF with PPO, the simpler DPO loss, and common failure modes such as reward hacking and sycophancy.

30 minIntermediateCode Examples

The Core Idea: Alignment teaches a model what people prefer. People compare two answers, a model learns from those comparisons, and the model is nudged toward the winners, either with a reward model and RLHF or directly with DPO.

  1. Beginner: Why a Smart Model Is Not Yet a Good Assistant

After pre-training and fine-tuning, an LLM can write fluent text. But fluent is not the same as helpful, honest and safe. A model may give a confident wrong answer, follow a harmful request, ramble, or flatter the user. Alignment is the work of steering a model toward what people actually want.

The hard part is that good is difficult to write down as a rule. It is much easier for a person to look at two answers and say which one is better. Almost all modern alignment methods are built on that simple idea: learning from preferences.

HelpfulDoes the job
Answers the real question clearly and completely.
HonestTells the truth
Admits uncertainty instead of inventing facts.
HarmlessAvoids harm
Declines dangerous requests politely.

Analogy: Pre-training builds a very knowledgeable student. Alignment is the mentoring that teaches the student how to behave in front of real people.

  1. Beginner: Collecting Human Preferences

The raw material for alignment is a preference dataset. For each prompt, the model writes two or more answers, and a person (or sometimes another AI) marks which one is better:

PromptAnswer AAnswer BBetter
How do I boil an egg?Put it in boiling water for 8 to 10 minutes, then cool it in cold water.Just google it.A (chosen)
Is the Earth flat?No. Satellite photos, ship horizons and flights all show it is round.Some people say yes!A (chosen)

Each pair gives a chosen answer and a rejected answer. Thousands to hundreds of thousands of such pairs are used. The quality and diversity of the labelers matters a lot, because the model will learn their idea of good.

AI feedback: Collecting human labels is slow and costly, so some methods let a strong model judge the answers using a written list of principles (for example Constitutional AI). This is often called RLAIF.

  1. Medium: RLHF, Teaching a Reward Model

RLHF (Reinforcement Learning from Human Feedback) was the first widely used recipe. It has three stages:

1
SFT: Fine-tune the base model on good example conversations.
2
Reward model: Train a second model that gives any answer a score reflecting how much humans would like it.
3
Reinforcement learning: Let the language model write answers, score them with the reward model, and nudge the model toward higher scores (usually with an algorithm called PPO).

The reward model learns from pairs using the Bradley-Terry idea: the probability that answer A beats answer B depends on the difference of their scores.

P(A≻B)=σ(r(A)−r(B))P(A \succ B) = \sigma\big(r(A) - r(B)\big)

The leash: If the language model could chase reward freely it would find tricks that fool the reward model (called reward hacking). So training adds a KL penalty that keeps the new model close to the original one.

  1. Medium: DPO, Skipping the Reward Model

RLHF works, but it is complex: three models, an unstable reinforcement-learning loop and many settings to tune. DPO (Direct Preference Optimization) reaches a similar goal with one simple loss that is trained like normal fine-tuning:

LDPO=−log⁡σ(β[(log⁡π(yw)−log⁡πref(yw))−(log⁡π(yl)−log⁡πref(yl))])\mathcal{L}_{DPO} = -\log \sigma\Big(\beta\big[(\log \pi(y_w) - \log \pi_{ref}(y_w)) - (\log \pi(y_l) - \log \pi_{ref}(y_l))\big]\Big)

In plain words: compare the model with a frozen reference copy. Raise the probability of the chosen answer ywy_w relative to the reference, lower the probability of the rejected answer yly_l, and the knob β\beta controls how far the model may drift.

RLHF (PPO)DPO
Models neededPolicy, reference, reward model (and a value model)Policy and a frozen reference
TrainingReinforcement-learning loop, sampling neededPlain supervised-style training on pairs
ComplexityHigh and sometimes unstableMuch simpler
Used whenMaximum control, large teamsQuick, stable preference tuning

  1. Advanced: Failure Modes and Safety Limits

Alignment is not a finished problem. Models trained on human approval can pick up unwanted habits:

ProblemWhat it looks likeCommon mitigation
Reward hackingLonger or more confident answers that score well but are not betterKL penalty, better reward models, human spot checks
SycophancyAgreeing with the user even when the user is wrongPreference data that rewards polite disagreement
Over-refusalRefusing harmless requestsBalanced data with both safe and helpful examples
Preference noiseLabelers disagree with each otherClear guidelines, several labelers, quality audits
JailbreaksPrompts that trick the model into ignoring its trainingRed-teaming and extra safety layers

Remember: Alignment shapes behavior but does not make a model perfect. Real systems combine aligned models with testing, monitoring and extra safeguards.

  1. Visual Explanation: Two Roads to an Aligned Model

Both roads start from the same preference pairs. The figure compares the steps each one needs, then shows where the data comes from.

Part 1 · RLHF vs. DPOSame goal, different road
Path A · RLHF3 stages
Humans compare answers
↓
Reward model learns the scores
↓
PPO nudges the policy (with a KL leash)
↓
Aligned model (complex, flexible)
Path B · DPO1 stage
Humans compare answers
↓
Direct loss on chosen vs. rejected
↓
Frozen reference keeps the model close
↓
Aligned model (simple, stable)
Part 2 · The Preference DataEverything starts with pairs
Step 1 · Prompt
A question
Sampled from real or synthetic requests.
Step 2 · Two answers
A and B
Written by the model being trained.
Step 3 · A label
Chosen vs. rejected
By humans, or by an AI judge with written principles.
Click to see how the DPO loss behavesCompare ▼
If the policy still equals the reference the loss is about 0.693. When the policy likes the chosen answer more, the loss falls. When it prefers the rejected answer, the loss rises above 0.693. The code below prints these exact cases.

Read Part 1 left to right: RLHF has one extra learned model (the reward model), DPO does not. Part 2 shows the shared starting point: pairs of answers with a chosen winner. Open the bar at the bottom to see how the DPO loss moves.

  1. Python Implementation: A Reward Model and the DPO Loss

This script does two things from scratch. First it trains a tiny reward model from noisy human choices between pairs of answers. Then it computes the DPO loss for four different situations.

preference_learning.pyPython 3.10+ · NumPy only
import numpy as np def sigmoid(x):    return 1 / (1 + np.exp(-x)) rng = np.random.default_rng(0) # 1. Reward model: learn what humans prefer from pairs of answers (Bradley-Terry)# Each answer is described by 2 features: [helpfulness, rudeness]w_true = np.array([2.0, -3.0])                   # humans like helpful, dislike rudeA = rng.normal(size=(500, 2))                    # answer A featuresB = rng.normal(size=(500, 2))                    # answer B featuresprefer_A = (sigmoid((A - B) @ w_true) > rng.random(500)).astype(float)   # noisy human choice w = np.zeros(2)for step in range(500):    p = sigmoid((A - B) @ w)                     # model's chance that A beats B    grad = (A - B).T @ (p - prefer_A) / len(p)   # gradient of the logistic loss    w -= 0.5 * gradprint("Learned reward weights [helpful, rude]:", np.round(w, 2), "| humans secretly used", w_true)agree = np.mean((sigmoid((A - B) @ w) > 0.5) == (prefer_A == 1))print("Reward model agrees with human choices:", round(float(agree), 3)) # 2. DPO loss: no reward model needed, only log-probabilities of the two answersdef dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta=0.1):    margin = beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected))    return -np.log(sigmoid(margin)) ref_c, ref_r = -12.0, -12.0                      # reference (frozen SFT model) log-probsfor name, pc, pr in [("unchanged policy", -12.0, -12.0),                     ("likes chosen more", -10.0, -12.0),                     ("likes chosen a lot", -6.0, -18.0),                     ("prefers the REJECTED", -14.0, -9.0)]:    print(name, "-> DPO loss", round(float(dpo_loss(pc, pr, ref_c, ref_r)), 3))
EXPECTED OUTPUT (your numbers may differ slightly)
Learned reward weights [helpful, rude]: [ 1.72 -2.68] | humans secretly used [ 2. -3.]
Reward model agrees with human choices: 0.874
unchanged policy -> DPO loss 0.693
likes chosen more -> DPO loss 0.598
likes chosen a lot -> DPO loss 0.263
prefers the REJECTED -> DPO loss 0.974

The learned weights point the same way as the hidden human preferences (helpful is good, rude is bad), and the model agrees with most choices even though the humans were noisy. For DPO, the loss starts at 0.693 when nothing has changed, shrinks as the policy favors the chosen answer, and grows when it favors the rejected one.

Interactive Knowledge Check

Question 1: What data does preference-based alignment use?
Question 2: What does DPO do differently from RLHF?

Key Points

✓Alignment steers a capable model toward answers that are helpful, honest and harmless.
✓It is built on preference pairs: a chosen answer and a rejected answer for the same prompt.
✓RLHF = SFT, then a reward model, then reinforcement learning with a KL penalty to stay close to the original model.
✓DPO gets a similar result with one simple loss against a frozen reference model.
✓Typical failures are reward hacking, sycophancy, over-refusal and noisy labels.
✓Alignment improves behavior but is not a guarantee, so real systems add testing and safeguards.

The Big Picture

Alignment in One Line

Compare answers → learn what people prefer → nudge the model toward it (but not too far)

Pre-training gives knowledge, fine-tuning gives format, and alignment gives judgment about what a good answer is. All three together make a usable assistant.

Up Next: A trained, aligned model still has to choose each next word. In Text Generation and Sampling you will see how temperature, top-k and top-p turn probabilities into text.