Introduction

Stochastic Gradient Descent (SGD) is a popular optimization algorithm used to minimize the loss function in machine learning models. Despite its simplicity and effectiveness, practitioners often encounter a perplexing phenomenon: after several iterations, the loss starts increasing instead of decreasing. This departure from expected behavior requires careful examination. In this article, we explore the reasons behind this phenomenon and suggest potential remedies.

Understanding SGD

SGD is an iterative method for optimizing an objective function, typically the loss function in machine learning tasks. Unlike batch gradient descent that uses the entire dataset, SGD updates the model parameters using a few samples. This makes SGD faster and more suitable for large datasets, albeit with increased variance in the updates.

Mathematical Formulation

The update rule for SGD is given by:

$$w^{(t+1)} = w^{(t)} - \eta \nabla L(w^{(t)}; x^{(t)}, y^{(t)})$$

where: • $w^{(t)}$ are the model parameters at iteration $t$. • $\eta$ is the learning rate. • $\nabla L(w^{(t)}; x^{(t)}, y^{(t)})$ is the gradient of the loss function with respect to $w^{(t)}$, evaluated at training sample $(x^{(t)}, y^{(t)})$.

Why Loss

Increases After Some Iterations

1. Learning Rate Issues

The learning rate, $\eta$, plays a crucial role in the convergence of SGD. If $\eta$ is too large, the algorithm may overshoot the minimum, causing the loss to increase. Conversely, a very small $\eta$ can lead to slow convergence and getting stuck in saddle points.

Solution:

• Use a learning rate scheduler to adjust $\eta$ over time. • Implement adaptive learning rates like Adam which adjust the learning rate based on past gradients.

2. Non-convex Loss

Surfaces

In complex models, such as deep neural networks, the loss surface may be highly non-convex with multiple local minima and saddle points. SGD can oscillate or get trapped in these regions, leading to an increase in loss.

Solution:

• Utilize momentum or other SGD variants that help in navigating complex surfaces. • Properly initialize weights to avoid flat or poor regions of the loss surface.

3. Poor Data Representation

Batch diversity is important in ensuring good convergence. If the mini-batches are not representative of the entire dataset, SGD updates could lead to poor parameter adjustments, increasing loss.

Solution:

• Shuffle data before creating mini-batches to ensure diversity. • Use techniques like batch normalization to stabilize learning.

Example Scenario

Consider training a neural network to classify images. If the learning rate is improperly set and batches are not shuffled, the network may initially find a good descent path, but diverge as it hits plateau regions or experiences large parameter jumps. This scenario illustrates how intertwined SGD dynamics are with hyperparameters and data handling.

Experiment Data

Let's summarize an experiment comparing the behavior of SGD with different learning schedules:

This table demonstrates how a decaying learning rate or adaptive methods like Adam can result in a consistently decreasing loss, whereas a fixed learning rate may lead to divergence after some iterations.

Additional Considerations

Robust Loss

Functions

Choosing an appropriate loss function can mitigate the effects of outliers and robustness issues inherent in SGD. Loss functions like Huber or modified versions tailored to the problem can provide smoother convergence.

Regularizations

Introduce L1, L2, or dropout regularization techniques to prevent the model from overfitting, which can also contribute to fluctuating loss behavior.

Monitoring and Early Stopping

Continuously monitor loss during training to detect and act on divergence quickly. Implement early stopping to halt training when improvements stall.

Conclusion

While SGD is a powerful optimization tool, its performance is closely tied to careful hyperparameter tuning and data handling strategies. By understanding the underlying causes of increasing loss during training, practitioners can effectively manage and mitigate these issues, ensuring better model performance and stability.

Through experimentation and adaptation, SGD becomes a robust means to train sophisticated machine learning models in practical settings.