In the world of machine learning, a typical challenge arises when training a model: the phenomenon where training loss increases after a certain number of epochs. In this context, we will explore why this may occur, using technical examples to illustrate the underlying concepts. We will consider potential causes, diagnostic methods, and mitigation strategies. This phenomenon is pivotal as it can affect the generalization and convergence of models.

Understanding Training `Loss`

Training loss is a measure of how well a machine learning model fits the training data. It quantifies the error or difference between predicted and actual output. Typically, during the initial phases of training, the model continuously minimizes this loss through optimization algorithms such as stochastic gradient descent (SGD).

Causes of Increasing Training `Loss` Post 12 Epochs

1. Learning Rate Issues:
   • High Learning Rate: A high learning rate can cause the model to overshoot the optimal weights, resulting in increased training loss. For instance, if the learning rate is set at `0.1`, it might be beneficial to reduce it to `0.01` after observing an increase in loss.
2. Overfitting:
   • After a certain point, if the model becomes too complex, it starts memorizing the training data rather than learning to generalize better to unseen data. This overfitting can cause fluctuations in training loss.
3. Suboptimal Batch Sizes:
   • Incorrect choice of batch sizes can destabilize training, resulting in oscillations in the training loss curve. Smaller batch sizes lead to more noise and variance during updates, while larger ones might lead to slower learning.
4. Internal Covariate Shift:
   • Shifts in input data distribution can lead to increased training loss. This occurs if earlier layers change weights significantly, disrupting subsequent layers' learning. Batch normalization can help mitigate this issue.
5. Model Instability:
   • Some models, particularly deep models, can be unstable and exhibit oscillations due to their architecture or initialization.

Example: Training a Convolutional Neural Network (CNN)

Consider training a CNN on the CIFAR-10 dataset for image classification. After 12 epochs, suppose the training loss starts to increase from an initially decreasing trend.

• Before epoch 12, the model learns effectively, reducing training loss.
• At epoch 12, loss begins to rise due to potential overfitting, high learning rate, or other causes mentioned above.
  • Use learning rate scheduling. Tools like learning rate annealing and adaptive optimizers (e.g., Adam) dynamically adjust the learning rate.
  • Monitor both training and validation losses. If both increase, it indicates a fundamental issue, such as inappropriate learning rate or model divergence.
  • Visualize loss curves to identify patterns that suggest instability or overfitting.
  • Employ a learning rate scheduler to reduce the learning rate upon detecting increased loss.
  • Use techniques like dropout, `L1`/`L2` regularization to reduce overfitting.
  • Implement early stopping to halt training when performance starts to degrade.
  • Augment the training dataset to increase variability and reduce overfitting.