Introduction

Batch normalization is a technique that has become fundamental in training deep learning models, particularly convolutional neural networks (CNNs). By normalizing the inputs of each layer to have zero mean and unit variance, it stabilizes the learning process and reduces the number of training epochs required to train deep networks. But regardless of its benefits, there's often ambiguity about where to correctly apply batch normalization in a standard CNN architecture. This article will discuss the technical considerations, examples, and common practices for applying batch normalization in CNNs.

Overview of Batch Normalization

Batch normalization works by addressing the internal covariate shift problem, which refers to the changes in the distribution of network activations due to updating weights. By normalizing inputs before they pass through the next layer, it ensures that inputs to activations are consistent across mini-batches.

Mathematical Formulation

For a given mini-batch, the batch normalization is calculated as follows:

1. Mini-batch mean:\
   $$\mu_B = \frac{1}{m} \sum_{i=1}^{m} x_i$$
2. Mini-batch variance:\
   $$\sigma_B^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu_B)^2$$
3. Normalize:\
   $$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$
4. Scale and shift:\
   $$y_i = \gamma \hat{x}_i + \beta$$

Where $x_i$ is the input, $\gamma$ and $\beta$ are learnable parameters, and $\epsilon$ is a small constant to prevent division by zero.

Where to Apply Batch Normalization

Common Practice

1. After Convolution and Before Activation: • A typical application is to place batch normalization after a convolutional (Conv) layer and before a nonlinear activation function like ReLU. This order stabilizes the inputs to activation functions, making the network less sensitive to weight initialization and learning rates.
   • Similar to convolutional layers, batch normalization can also be applied after fully connected (Dense) layers, generally before applying activations. • Output Layers: Typically, batch normalization is not applied after the final output layer, especially when the network is performing regression. For classification tasks, it may inadvertently affect softmax outputs. • Small Mini-batches: If the mini-batch size is very small, the mean and variance estimates may be unreliable, in which case other normalization techniques like Layer Normalization might be more appropriate.