Introduction

When working with machine learning models, particularly neural networks, one critical issue that can arise is overfitting. Overfitting occurs when a model learns not just the underlying distribution of the training data but also the noise, leading it to perform poorly on unseen data. In this article, we'll explore how to avoid overfitting on a simple feedforward network, providing technical explanations and offering practical solutions.

Understanding Overfitting

What is Overfitting?

Overfitting is a modeling error that occurs when a machine learning model captures the noise of the training data instead of its actual distribution. This results in a model that performs well on the training data but poorly on new, unseen data.

Symptoms of Overfitting

• High accuracy on training data but low accuracy on validation or test data. • Increasing gap between training and validation loss/accuracy over epochs.

Strategies to Avoid Overfitting

We can employ various techniques to mitigate overfitting in a feedforward neural network.

1. Regularization

Regularization adds a penalty to the loss function to discourage overly complex models. There are two common types:

• L1 Regularization: Adds a penalty equal to the absolute value of the magnitude of coefficients. It can reduce some weights to zero, effectively performing feature selection.

• L2 Regularization: Adds a penalty equal to the square of the magnitude of coefficients, also known as weight decay. This encourages smaller network weights.

Mathematical Formulation for L2 Regularization:

$$\mathcal{L}(\theta) = \frac{1}{N} \sum\_{i=1}^{N} \text{Loss}(y\_i, \hat{y}*i) + \lambda \sum*{j=1}^{M} \theta\_j^2$$

Where $\lambda$ is the regularization parameter, $\theta$ represents weights, and $M$ is the total number of weights.

2. Dropout

Dropout is a simple yet powerful technique. It randomly "drops" or sets some neurons to zero during training, which prevents neurons from co-adapting too much.

Example Implementation: