The Core Idea: Training a model means finding the parameter values that make its error as small as possible. Gradient descent does this without any magic formula: it checks which direction makes the error go down fastest, takes a small step that way, and repeats until it reaches the bottom.
- Beginner: The Blindfolded Hiker (The Core Intuition)
Imagine you are standing on a foggy mountain, blindfolded, and your goal is to reach the lowest valley. You cannot see the whole landscape. All you can do is feel the slope of the ground under your feet. If the ground tilts downward to your right, you take a small step to the right. Then you feel the slope again and repeat. Step by step, you walk downhill until the ground feels flat. That is gradient descent.
For very simple models there is a direct formula for the best parameters. But a modern neural network has millions or billions of parameters and a bumpy, complicated loss surface. No direct formula exists, so we use a patient step-by-step search instead. Gradient descent is that search, and it powers almost every model you will meet in this course.
- Beginner: The Loss Landscape and What a Gradient Really Is
A loss function turns the model's parameters into a single number that measures its error. For example, in linear regression we often use the average squared error between predictions and true values. Plotting against gives a curve (for one parameter) or a surface (for many parameters). We want the lowest point.
The gradient is the derivative of the loss with respect to the parameters. With one parameter it is just the slope of the curve at the current point:
With many parameters, the gradient is a list of slopes, one per parameter. Together they point in the direction of the steepest uphill. So gradient descent always walks in the opposite direction.
- Medium: The Update Rule and a Worked Example
Every step of gradient descent uses one simple rule. Subtract the gradient, scaled by a small number called the learning rate :
Let us run it by hand on a tiny loss function, . Its lowest point is clearly at , and its gradient is . Start at with :
| Step | Current | Gradient | Loss | New |
|---|---|---|---|---|
| 0 | 0.0000 | -8.000 | 16.000 | 0.8000 |
| 1 | 0.8000 | -6.400 | 10.240 | 1.4400 |
| 2 | 1.4400 | -5.120 | 6.554 | 1.9520 |
| 3 | 1.9520 | -4.096 | 4.194 | 2.3616 |
Notice two things. The loss shrinks every step, and the steps get smaller as we approach the minimum, because the slope flattens out. For this function the remaining distance to is multiplied by at every step, so we get closer and closer without overshooting.
- Medium: The Learning Rate Decides Everything
The learning rate is the size of each step, and it is the single most important setting in gradient descent. On the same function the error is multiplied by per step, which makes the three classic cases easy to see:
Practical tip: If your loss goes up, becomes nan, or jumps around wildly, lower the learning rate first. If the loss falls but extremely slowly, try raising it a little. Later you will learn learning-rate schedules that change during training.
- Advanced: Batch, Stochastic and Mini-Batch Gradient Descent
Real datasets can have millions of rows. The loss gradient is an average over the training examples, so there is a choice: how many examples should we use to estimate the gradient before taking each step?
| Variant | Examples per step | Strength | Weakness |
|---|---|---|---|
| Batch | All of them | Accurate, smooth path | Slow and memory-heavy on big data |
| Stochastic (SGD) | Exactly 1 | Very fast updates | Noisy, zig-zag path |
| Mini-batch | A small group, such as 32 or 256 | Good balance, fits GPUs well | Needs a batch size choice |
Almost all deep learning uses mini-batch gradient descent. The small amount of noise it adds is usually helpful, because it can shake the parameters out of shallow traps. Modern optimizers such as momentum, RMSProp and Adam are improved versions of this same idea: they still follow the gradient, but they adapt the step size and direction using the history of past gradients.
The bowl-shaped loss in our example has one minimum, so gradient descent always finds it. Neural network losses are bumpier. Gradient descent can slow down on flat plateaus and saddle points (flat in one direction, curved in another). In practice, for large networks, the noise of mini-batches and smarter optimizers make this much less of a problem than it first sounds.
- Visual Explanation: The Descent Loop and the Learning Rate Explorer
The figure below has two parts. Part 1 shows the whole algorithm as a loop with a stopping check. Part 2 lets you pick a learning rate and see the first three steps on starting from .
Click to see why the steps shrink near the minimumExplain ▼
How to read it: follow Phase A top to bottom, then ask the Phase B question. If the answer is no, the loop starts again. In Part 2, compare the three buttons to see how the same loop behaves with different step sizes.
- Python Implementation: Linear Regression Trained With Gradient Descent From Scratch
Here is the algorithm from scratch with NumPy. It fits a line to noisy data by repeating the four steps of the loop: predict, measure the loss, compute the gradients, and update. At the end we compare the result with the exact least-squares solution from np.polyfit.
# Linear regression trained with gradient descent, from scratchimport numpy as np # 1. Make data that follows y = 3x + 4 plus some noiserng = np.random.default_rng(42)x = rng.uniform(0, 10, size=100)y = 3.0 * x + 4.0 + rng.normal(0, 1.0, size=100) def gradient_descent(x, y, lr=0.01, steps=2000): n = len(y) w, b = 0.0, 0.0 # start from zero history = [] for step in range(steps): # 2. Predict and measure the error error = (w * x + b) - y loss = np.mean(error ** 2) history.append(loss) # 3. Gradients of the mean squared error dw = (2 / n) * np.sum(error * x) db = (2 / n) * np.sum(error) # 4. Step downhill w = w - lr * dw b = b - lr * db if step % 400 == 0: print("step", step, "| loss", round(loss, 3), "| w", round(w, 3), "| b", round(b, 3)) return w, b, history w, b, history = gradient_descent(x, y, lr=0.01, steps=2000)print("Gradient descent -> w =", round(w, 3), ", b =", round(b, 3)) # 5. Check against the exact least-squares answerbest_w, best_b = np.polyfit(x, y, 1)print("Exact solution -> w =", round(best_w, 3), ", b =", round(best_b, 3))Gradient descent recovers almost exactly the same line as the exact formula. Try changing lr to 0.05 (the loss will blow up because the step is too big for this data) and to 0.001 (it will learn, but far too slowly in 2000 steps).
Interactive Knowledge Check
Key Points
The Big Picture
Measure the error → Find the downhill direction → Take a small step → Repeat until the loss stops falling
Every time a model "learns", some version of this loop is running. The only things that change are the model, the loss function, and how cleverly the step is chosen.
Up Next: You now know how a single step is taken. In Backpropagation you will see how the gradients of a deep neural network, with many layers, are computed efficiently so this same loop can train it.