Gradient Descent

Learn how gradient descent finds the lowest point of a loss function step by step—from the blindfolded-hiker intuition and the update rule to learning rates, batch vs. mini-batch training, and a from-scratch NumPy implementation.

30 minBeginnerCode Examples

The Core Idea: Training a model means finding the parameter values that make its error as small as possible. Gradient descent does this without any magic formula: it checks which direction makes the error go down fastest, takes a small step that way, and repeats until it reaches the bottom.

  1. Beginner: The Blindfolded Hiker (The Core Intuition)

Imagine you are standing on a foggy mountain, blindfolded, and your goal is to reach the lowest valley. You cannot see the whole landscape. All you can do is feel the slope of the ground under your feet. If the ground tilts downward to your right, you take a small step to the right. Then you feel the slope again and repeat. Step by step, you walk downhill until the ground feels flat. That is gradient descent.

The MountainLoss Function
Meaning: How wrong the model is for the current parameter values. High = wrong, low = good.
Your PositionParameters
Meaning: The weights and biases the model can change. Moving means changing these numbers.
The SlopeGradient
Meaning: Tells you which direction is uphill and how steep it is, so you can walk the other way.
Why not just solve it with a formula?

For very simple models there is a direct formula for the best parameters. But a modern neural network has millions or billions of parameters and a bumpy, complicated loss surface. No direct formula exists, so we use a patient step-by-step search instead. Gradient descent is that search, and it powers almost every model you will meet in this course.

  1. Beginner: The Loss Landscape and What a Gradient Really Is

A loss function J(w)J(w) turns the model's parameters into a single number that measures its error. For example, in linear regression we often use the average squared error between predictions and true values. Plotting J(w)J(w) against ww gives a curve (for one parameter) or a surface (for many parameters). We want the lowest point.

The gradient is the derivative of the loss with respect to the parameters. With one parameter it is just the slope of the curve at the current point:

Slope is positive
The loss rises as ww increases, so the way down is to decrease ww.
Slope is negative
The loss falls as ww increases, so the way down is to increase ww.
Slope is zero
The ground is flat. We have reached a minimum (or occasionally a flat spot).

With many parameters, the gradient ∇J\nabla J is a list of slopes, one per parameter. Together they point in the direction of the steepest uphill. So gradient descent always walks in the opposite direction.

  1. Medium: The Update Rule and a Worked Example

Every step of gradient descent uses one simple rule. Subtract the gradient, scaled by a small number called the learning rate α\alpha:

wnew=wold−α⋅∂J∂ww_{new} = w_{old} - \alpha \cdot \frac{\partial J}{\partial w}

Let us run it by hand on a tiny loss function, J(w)=(w−4)2J(w) = (w - 4)^2. Its lowest point is clearly at w=4w = 4, and its gradient is ∂J∂w=2(w−4)\frac{\partial J}{\partial w} = 2(w - 4). Start at w=0w = 0 with α=0.1\alpha = 0.1:

StepCurrent wwGradient 2(w−4)2(w-4)Loss J(w)J(w)New ww
00.0000-8.00016.0000.8000
10.8000-6.40010.2401.4400
21.4400-5.1206.5541.9520
31.9520-4.0964.1942.3616

Notice two things. The loss shrinks every step, and the steps get smaller as we approach the minimum, because the slope flattens out. For this function the remaining distance to w=4w = 4 is multiplied by (1−2α)=0.8(1 - 2\alpha) = 0.8 at every step, so we get closer and closer without overshooting.

  1. Medium: The Learning Rate Decides Everything

The learning rate α\alpha is the size of each step, and it is the single most important setting in gradient descent. On the same function J(w)=(w−4)2J(w) = (w-4)^2 the error is multiplied by (1−2α)(1 - 2\alpha) per step, which makes the three classic cases easy to see:

Too Smallα=0.01\alpha = 0.01
Factor per step: 0.980.98
Behavior: Safe but painfully slow. Thousands of steps may be needed.
Risk: Wastes compute and can stall on flat regions.
Just Rightα=0.1\alpha = 0.1
Factor per step: 0.800.80
Behavior: Smooth, steady approach to the minimum.
Risk: Low. This is the goal.
Too Largeα=1.1\alpha = 1.1
Factor per step: −1.2-1.2 (size grows)
Behavior: Each step overshoots the valley and lands higher up the other side.
Risk: The loss explodes. This is called divergence.

Practical tip: If your loss goes up, becomes nan, or jumps around wildly, lower the learning rate first. If the loss falls but extremely slowly, try raising it a little. Later you will learn learning-rate schedules that change α\alpha during training.

  1. Advanced: Batch, Stochastic and Mini-Batch Gradient Descent

Real datasets can have millions of rows. The loss gradient is an average over the training examples, so there is a choice: how many examples should we use to estimate the gradient before taking each step?

VariantExamples per stepStrengthWeakness
BatchAll of themAccurate, smooth pathSlow and memory-heavy on big data
Stochastic (SGD)Exactly 1Very fast updatesNoisy, zig-zag path
Mini-batchA small group, such as 32 or 256Good balance, fits GPUs wellNeeds a batch size choice

Almost all deep learning uses mini-batch gradient descent. The small amount of noise it adds is usually helpful, because it can shake the parameters out of shallow traps. Modern optimizers such as momentum, RMSProp and Adam are improved versions of this same idea: they still follow the gradient, but they adapt the step size and direction using the history of past gradients.

Local minima, saddle points and plateaus

The bowl-shaped loss in our example has one minimum, so gradient descent always finds it. Neural network losses are bumpier. Gradient descent can slow down on flat plateaus and saddle points (flat in one direction, curved in another). In practice, for large networks, the noise of mini-batches and smarter optimizers make this much less of a problem than it first sounds.

  1. Visual Explanation: The Descent Loop and the Learning Rate Explorer

The figure below has two parts. Part 1 shows the whole algorithm as a loop with a stopping check. Part 2 lets you pick a learning rate and see the first three steps on J(w)=(w−4)2J(w) = (w-4)^2 starting from w=0w = 0.

Part 1 · The Descent LoopWalk the loop until the slope is flat
Phase A · One StepRepeats
Start with random parameters
↓
Compute the loss
↓
Compute the gradient
↓
Update: w = w − α × gradient
Phase B · Stop CheckEvery step
Is the gradient close to zero?
↓
No → go back to Phase A
↓
Yes → minimum found, stop
Part 2 · Learning Rate ExplorerClick a button · the minimum is at w = 4
Result: the error is multiplied by (1 − 2α) every step
Click to see why the steps shrink near the minimumExplain ▼
Each step is the learning rate times the gradient. Near the bottom of the bowl the slope is almost flat, so the gradient is tiny and so is the step. With α = 0.1 the leftover distance to 4 shrinks by a factor of 0.8 each time: 4.0, 3.2, 2.56, 2.05, and so on.

How to read it: follow Phase A top to bottom, then ask the Phase B question. If the answer is no, the loop starts again. In Part 2, compare the three buttons to see how the same loop behaves with different step sizes.

  1. Python Implementation: Linear Regression Trained With Gradient Descent From Scratch

Here is the algorithm from scratch with NumPy. It fits a line y^=wx+b\hat{y} = wx + b to noisy data by repeating the four steps of the loop: predict, measure the loss, compute the gradients, and update. At the end we compare the result with the exact least-squares solution from np.polyfit.

gradient_descent_from_scratch.pyPython 3.10+ · NumPy only
# Linear regression trained with gradient descent, from scratchimport numpy as np # 1. Make data that follows y = 3x + 4 plus some noiserng = np.random.default_rng(42)x = rng.uniform(0, 10, size=100)y = 3.0 * x + 4.0 + rng.normal(0, 1.0, size=100) def gradient_descent(x, y, lr=0.01, steps=2000):    n = len(y)    w, b = 0.0, 0.0                      # start from zero    history = []     for step in range(steps):        # 2. Predict and measure the error        error = (w * x + b) - y        loss = np.mean(error ** 2)        history.append(loss)         # 3. Gradients of the mean squared error        dw = (2 / n) * np.sum(error * x)        db = (2 / n) * np.sum(error)         # 4. Step downhill        w = w - lr * dw        b = b - lr * db         if step % 400 == 0:            print("step", step, "| loss", round(loss, 3), "| w", round(w, 3), "| b", round(b, 3))     return w, b, history w, b, history = gradient_descent(x, y, lr=0.01, steps=2000)print("Gradient descent -> w =", round(w, 3), ", b =", round(b, 3)) # 5. Check against the exact least-squares answerbest_w, best_b = np.polyfit(x, y, 1)print("Exact solution   -> w =", round(best_w, 3), ", b =", round(best_b, 3))
EXPECTED OUTPUT (your numbers may differ slightly)
step 0 | loss 413.504 | w 2.255 | b 0.372
step 400 | loss 1.025 | w 3.09 | b 3.426
step 800 | loss 0.96 | w 3.02 | b 3.868
step 1200 | loss 0.959 | w 3.01 | b 3.936
step 1600 | loss 0.959 | w 3.008 | b 3.947
Gradient descent -> w = 3.008 , b = 3.948
Exact solution -> w = 3.008 , b = 3.949

Gradient descent recovers almost exactly the same line as the exact formula. Try changing lr to 0.05 (the loss will blow up because the step is too big for this data) and to 0.001 (it will learn, but far too slowly in 2000 steps).

Interactive Knowledge Check

Question 1: Your training loss keeps increasing and finally becomes nan. What should you try first?
Question 2: Which variant is used in almost all deep learning training?

Key Points

✓Gradient descent minimizes a loss function by repeatedly stepping in the opposite direction of the gradient.
✓The update rule is new = old - learning rate x gradient, applied to every parameter at each step.
✓The learning rate controls step size: too small is slow, too large overshoots and diverges.
✓The gradient shrinks near a minimum, so steps naturally get smaller as training converges.
✓Mini-batch gradient descent (32 to 256 examples per step) is the standard choice for deep learning.
✓Optimizers like momentum and Adam are refinements of the same idea, not replacements for it.

The Big Picture

Gradient Descent in One Line

Measure the error → Find the downhill direction → Take a small step → Repeat until the loss stops falling

Every time a model "learns", some version of this loop is running. The only things that change are the model, the loss function, and how cleverly the step is chosen.

Up Next: You now know how a single step is taken. In Backpropagation you will see how the gradients of a deep neural network, with many layers, are computed efficiently so this same loop can train it.