Introduction

Gradient Descent is the workhorse optimization algorithm in machine learning and deep learning. It minimizes a cost function by iteratively updating model parameters in the direction of the negative gradient. However, vanilla gradient descent faces several challenges: slow convergence on ill-conditioned problems, sensitivity to learning rate selection, and a tendency to get stuck in local minima or saddle points. To address these issues, several alternative and improved optimization algorithms have been developed. This article covers the most important ones with their mathematical formulations and practical trade-offs.

Alternatives to Gradient Descent

1. Stochastic Gradient Descent (SGD) and Mini-batch SGD

Description: While vanilla gradient descent uses the entire dataset to compute the gradient, SGD uses only one randomly sampled example per iteration. Mini-batch SGD uses a small subset (typically 32 to 512 examples).

Pros:

• Much faster per iteration since it processes less data.
• The noise from random sampling can help escape shallow local minima.

Cons:

• Higher variance in parameter updates, causing oscillation.
• Requires careful tuning of learning rate and batch size.

2. Momentum

Description: Momentum augments SGD by adding a fraction of the previous update to the current one. This acts like a ball rolling downhill, accumulating velocity in consistent directions and dampening oscillations.

Update rules:

$$v_t = \gamma v_{t-1} + \eta \nabla J(\theta_t)$$

$$\theta_t = \theta_{t-1} - v_t$$

where $\gamma$ is the momentum coefficient (typically 0.9) and $\eta$ is the learning rate.

Pros:

• Accelerates convergence in directions with consistent gradient.
• Dampens oscillations in directions with high curvature.

Cons:

• Introduces one additional hyperparameter ($\gamma$).

3. Nesterov Accelerated Gradient (NAG)

Description: NAG improves on momentum by computing the gradient at a "look-ahead" position rather than the current position. This anticipation allows the optimizer to correct its course before overshooting.

Update rules:

$$\theta_{temp} = \theta_{t-1} - \gamma v_{t-1}$$

$$v_t = \gamma v_{t-1} + \eta \nabla J(\theta_{temp})$$

$$\theta_t = \theta_{t-1} - v_t$$

Pros:

• Reduces the overshooting problem of standard momentum.
• Generally converges faster than momentum SGD.

Cons:

• Slightly more computationally expensive due to the extra gradient evaluation.

4. Adagrad

Description: Adagrad adapts the learning rate for each parameter individually, scaling it inversely by the accumulated squared gradients. Parameters that receive large gradients get smaller learning rates, and vice versa.

Update rule:

$$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_{t} + \epsilon}} \nabla J(\theta_t)$$

where $G_t$ is the diagonal matrix of accumulated squared gradients and $\epsilon$ is a small constant (around $10^{-8}$) for numerical stability.

Pros:

• Well-suited for sparse data (e.g., NLP tasks with large vocabularies).
• Eliminates the need to manually tune the learning rate.

Cons:

• The accumulated squared gradients grow monotonically, causing the effective learning rate to shrink to near zero over time. Training can stall.

5. RMSProp

Description: RMSProp addresses Adagrad's vanishing learning rate by replacing the cumulative sum with an exponentially decaying moving average of squared gradients.

Update rules:

$$E[g^2]_t = \rho \cdot E[g^2]_{t-1} + (1 - \rho) \cdot g_t^2$$

$$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}} \cdot g_t$$

where $\rho$ is the decay rate (typically 0.9).

Pros:

• Maintains a stable effective learning rate throughout training.
• Works well for non-stationary objectives (e.g., RNNs).

Cons:

• Requires selecting the decay rate hyperparameter.

6. Adam (Adaptive Moment Estimation)

Description: Adam combines the ideas of momentum (first moment) and RMSProp (second moment). It maintains exponentially decaying averages of both the gradient and the squared gradient, with bias correction for the initial time steps.

Update rules:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

Bias-corrected estimates:

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$$

$$\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

Parameter update:

$$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

Default hyperparameters from the original paper: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$.

Pros:

• Fast convergence in practice.
• Works well across a wide range of problems with minimal tuning.

Cons:

• More hyperparameters (though defaults work well in most cases).
• Can generalize worse than well-tuned SGD with momentum in some cases.

Comparison Table

Practical Guidance

• Start with Adam for most deep learning tasks. Its defaults ($\beta_1 = 0.9$, $\beta_2 = 0.999$, $\eta = 0.001$) work well without tuning.
• Switch to SGD with momentum for final training if you need the best generalization (common in computer vision with ResNets).
• Use Adagrad for sparse features like text classification with bag-of-words representations.
• Use RMSProp for recurrent neural networks where non-stationary objectives are common.

Summary

While vanilla gradient descent remains a foundational concept, modern deep learning relies on adaptive optimizers that adjust learning rates per parameter and incorporate momentum. Adam is the most popular default choice due to its robustness, while SGD with momentum remains competitive when carefully tuned. Understanding the mathematical foundations of each optimizer helps you diagnose training issues and select the right tool for your problem.