Training a neural network is an optimization problem: minimize a loss function by adjusting millions of parameters. The algorithm that does this, the optimizer, determines whether training succeeds, how fast it converges, and whether the result generalizes. Understanding the evolution from plain gradient descent to Adam is not just historical context; it explains why each component of Adam exists and what problem it solves.

Vanilla Gradient Descent

The simplest update rule: take a step in the direction of steepest descent of the loss.

Vanilla (batch) gradient descent computes the gradient over the entire training set before updating. On a dataset of 1 million examples, that means a single weight update requires processing every example, far too slow for practical training.

Stochastic Gradient Descent (SGD)

SGD approximates the true gradient using a single random example (or a small mini-batch):

The gradient estimate is noisy, each mini-batch gives a different estimate of the true gradient. But this noise is actually useful: it prevents the optimizer from settling into sharp local minima and introduces regularization. The key hyperparameter is the learning rate $\eta$: too high and the updates overshoot the minimum; too low and training takes forever.

The fundamental problem with SGD is that it uses the same learning rate for every parameter. In practice, some directions of parameter space are very steep (loss changes rapidly) while others are very flat (loss changes slowly). SGD either takes too large a step in the steep directions (overshooting) or too small a step in the flat directions (slow progress). This is the "narrow valley" problem.

Momentum

Momentum addresses SGD's tendency to oscillate. Instead of updating directly with the current gradient, maintain a running velocity vector that accumulates gradients:

With $\beta = 0.9$, the velocity is an exponential moving average of past gradients. The optimizer builds up speed in consistent directions (the "valley floor") while damping oscillations across the valley (where gradients alternate sign).

Intuitively: think of a ball rolling down a surface. Without momentum, the ball takes each step independently. It stops at every plateau. With momentum, the ball accumulates velocity and can roll through small plateaus and past noisy gradient directions.

1# Momentum in NumPy
2v = np.zeros_like(theta)
3for step in range(num_steps):
4    g = compute_gradient(theta)
5    v = 0.9 * v + 0.1 * g        # accumulate velocity
6    theta = theta - lr * v

RMSProp

Momentum solves oscillation but still uses the same learning rate for all parameters. RMSProp addresses this by maintaining a per-parameter running average of squared gradients and dividing each parameter's update by its own gradient magnitude:

Parameters with consistently large gradients get smaller effective learning rates (divided by large $\sqrt{s}$). Parameters with small gradients get larger effective learning rates (divided by small $\sqrt{s}$). This is adaptive learning rates per parameter, the optimizer automatically calibrates to the geometry of each parameter's loss surface.

1# RMSProp in NumPy
2s = np.zeros_like(theta)
3for step in range(num_steps):
4    g = compute_gradient(theta)
5    s = 0.999 * s + 0.001 * (g ** 2)   # running avg of squared gradients
6    theta = theta - lr * g / (np.sqrt(s) + 1e-8)