Gradient Descent: The Hill-Climbing Algorithm Behind Deep Learning

Neural networks are function approximators. Gradient descent is how they find those functions — by rolling downhill on a loss surface in million-dimensional space.

The loop

1. Forward pass — compute predictions
2. Loss function — measure error (MSE, cross-entropy)
3. Backprop — compute gradients via chain rule
4. Update — nudge weights opposite the gradient

for epoch in range(epochs):
    loss = compute_loss(model, batch)
    grads = autograd(loss, model.params)
    for param, grad in zip(model.params, grads):
        param -= learning_rate * grad

Repeat until loss plateaus or your cloud bill explodes.

Learning rate matters

Too high → oscillation or divergence. Too low → weeks of training.

Momentum

SGD with momentum accumulates velocity — overshoots small local minima, accelerates through flat regions:

v = β * v + gradient
param -= lr * v

Think ball rolling downhill, gaining inertia.

Adam and friends

Adam adapts per-parameter learning rates using running averages of gradient and squared gradient. Default optimizer for most deep learning — not always optimal, rarely catastrophic.

Local minima myth

In high dimensions, saddle points outnumber true local minima. Modern networks are overparameterized — many global or near-global solutions exist. The hard part is finding them fast, not escaping traps.

Takeaway

Every LLM, every classifier, every diffusion model — same core loop. Master gradient descent and backprop, and the rest of deep learning is architecture choices stacked on top.