What Gradient Descent Actually Does

Gradient descent gets introduced with a lot of notation before it gets explained with any intuition, and I think that ordering does real damage. You end up able to write the update rule — θ = θ - α∇L(θ) — without being able to say, in plain terms, what it’s doing or why it works. I want to go the other direction: start from the mechanical idea, and let the notation show up afterward as a shorthand for something you already understand.

The idea without the symbols

Say you’re trying to find the lowest point in a landscape, but you’re standing in fog and can only see the ground immediately around your feet. You can’t see the whole landscape at once — you only know, locally, which direction is downhill from where you’re standing right now.

A reasonable strategy: take a step in the steepest downhill direction you can feel, then re-assess from your new position, and repeat. Eventually — assuming the landscape is reasonably well-behaved — you end up somewhere low. Maybe not the very lowest point in the whole landscape, but somewhere locally low, which is often good enough.

That’s gradient descent. The “landscape” is the loss function — a measure of how wrong your model’s predictions are, as a function of its parameters. The “downhill direction” is the negative gradient of that loss with respect to the parameters. The “step” is the parameter update. The fog is the fact that you generally can’t compute the whole loss surface, only its local slope at your current parameter values.

Why the gradient points where it does

The part that took me longest to feel comfortable with wasn’t the algorithm — it’s simple once you see it — but why the gradient is the direction of steepest ascent in the first place, and why moving opposite to it is the locally optimal move.

The gradient of a function at a point is a vector of partial derivatives — how much the function’s output changes as you nudge each input independently. If you’re standing at some point and want to move in the direction that increases the function fastest, it turns out that direction is exactly the gradient vector, and this isn’t a coincidence or a definition — it falls out of the fact that a small step in any direction changes the function’s output by approximately the dot product of that step with the gradient. The dot product between two vectors is maximized when they point the same way. So the step direction that maximizes the increase in the function is the one that points the same way as the gradient. Move opposite to it, and you get the direction of steepest decrease.

Once that clicks, the update rule stops looking like an arbitrary formula and starts looking like the obvious move: at every point, ask “which direction makes the loss worse fastest,” and go the other way, scaled by how big a step you’re willing to take.

A minimal implementation

Here’s gradient descent doing the simplest thing it can do — finding the minimum of a one-dimensional function — with nothing hidden:

def gradient_descent(grad_fn, start, learning_rate=0.1, steps=50):
    x = start
    history = [x]
    for _ in range(steps):
        grad = grad_fn(x)
        x = x - learning_rate * grad
        history.append(x)
    return x, history

# Minimize f(x) = (x - 3) ** 2, whose minimum is at x = 3
# The gradient of f is f'(x) = 2 * (x - 3)
grad_fn = lambda x: 2 * (x - 3)

result, path = gradient_descent(grad_fn, start=-10.0)
print(result)  # converges toward 3.0

Every piece of that maps directly onto the fog analogy. grad_fn(x) is feeling the slope under your feet. learning_rate is your step size. The loop is “take a step, reassess, repeat.” There’s no matrix calculus here because there’s only one parameter, but scaling this up to a network with millions of parameters doesn’t change the idea — it just means the gradient is a much longer vector, one entry per parameter, and each entry says how much nudging that particular parameter would change the loss, holding the others fixed.

Where the interesting failure modes live

Once you have the mechanical picture, the classic gradient descent problems stop being abstract warnings and start being predictable consequences.

The learning rate is a step size, so it can overshoot. If your step is too large, you can step clean over the minimum and land somewhere worse than where you started, then overshoot back the other way on the next step. Push the learning rate too high in the example above and instead of smoothly converging to 3, x oscillates with growing amplitude and diverges. This isn’t a special failure unique to neural networks — it’s what happens any time you take too large a step down a slope that curves back up.

A learning rate that’s too small isn’t wrong, just slow. You’ll still converge, in principle, but it might take far more steps than is practical, which in a real training run means far more compute and wall-clock time than you have.

Local minima are a real limitation, not a hypothetical one. Because you can only see the slope where you’re standing, gradient descent has no way to know if there’s a much better minimum somewhere else in the landscape, over a hill it isn’t inclined to climb. For genuinely non-convex loss surfaces — which most interesting ones are — this is a real constraint, not just a theoretical caveat in a textbook.

Flat regions slow everything down for a reason that has nothing to do with being “close” to an answer. If the slope near your current position is close to zero, your steps get tiny, even if you’re nowhere near the actual minimum. This is part of why techniques like momentum exist — momentum effectively lets you carry some velocity through flat regions instead of slowing to a crawl every time the local gradient happens to be small.

Why the fancier optimizers are still doing this

Once the plain version is intuitive, the fancier optimizers — momentum, RMSprop, Adam — stop feeling like separate algorithms and start looking like the same basic idea with extra bookkeeping. Momentum adds a running average of past gradients so the parameter update carries some memory of the direction it’s been moving, which smooths out the oscillation problem. Adam additionally tracks a running estimate of the gradient’s variance and uses it to scale the effective learning rate per parameter, so parameters with noisy or wildly varying gradients get smaller, more cautious steps, and parameters with small, consistent gradients get larger ones.

None of that changes the fundamental move: look at the local slope, take a step against it, repeat. The elaborations exist to make that basic move more stable and efficient at scale, not to replace it with something conceptually different.

Why bother with the fog analogy

I could have opened with the formal definition and it would have been correct. But correctness and understanding aren’t the same thing, and for me, the formal definition on its own never produced the second one. What did was building the mechanical picture first — the fog, the local slope, the step — and then noticing that the formula is just a compact way of writing exactly that. Once the picture is solid, the notation stops being something to memorize and becomes something to recognize.