Introduction

In the world of deep learning, choosing the right initialization method for neural network weights is crucial to ensure faster convergence and better performance. Two popular initialization strategies are Xavier (or Glorot) initialization and He (or He_normal) initialization. These methods have become standard practices in training deep neural networks, such as convolutional neural networks (CNNs) and fully connected networks. This article explores the differences between Xavier and He_normal initialization, providing technical explanations and examples.

Understanding Weight Initialization

Weight initialization is a technique for setting the initial weights of a neural network before the training begins. The choice of initialization can significantly impact the speed of convergence and the final performance of the network.

Why is Initialization Important?

• Avoid Vanishing/Exploding Gradients: Poor initialization can cause gradients to vanish (become too small) or explode (become too large), obstructing the learning process. • Symmetry Breaking: Proper initialization ensures that different neurons start by learning different things.

Xavier Initialization

Xavier initialization, named after its developer Xavier Glorot, is designed to maintain the variance of activations and backpropagated gradients across layers. It is particularly suited for networks using the sigmoid or hyperbolic tangent (tanh) activation functions.

Formula

For a layer with `n_{input}` incoming connections (or fan-in), the weights are initialized as follows:

• Uniform distribution: $$W \sim \text{U}\left(-\frac{\sqrt{6}}{\sqrt{n\_{input} + n\_{output}}}, \frac{\sqrt{6}}{\sqrt{n\_{input} + n\_{output}}}\right)$$

• Normal distribution: $$W \sim \mathcal{N}\left(0, \frac{1}{\sqrt{n\_{input} + n\_{output}}}\right)$$

Use Case

Xavier initialization is generally used for symmetric activation functions such as sigmoid or tanh due to its derivation, which assumes symmetry.

He_normal Initialization

He_normal initialization, introduced by He et al., is optimized for activation functions like Rectified Linear Units (ReLU). It considers the rectifier's properties, allowing better flow of gradients throughout the neural network.

Formula

For a layer with `n_{input}` incoming connections, the weights are initialized as follows:

• Normal distribution: $$W \sim \mathcal{N}\left(0, \frac{2}{n\_{input}}\right)$$

Use Case

He_normal initialization is particularly effective for ReLU and its variants (e.g., Leaky ReLU, Parametric ReLU).

Comparison of Xavier and He_normal Initialization

Practical Example

Let's illustrate the impact of both initialization methods using a simple neural network trained on the MNIST dataset.

Setup

Assume a simple feedforward network with:

• Input layer: 784 units (28x28 pixels) • One hidden layer: 128 units • Output layer: 10 units (one for each class)

Code Snippet

Below is a Python snippet showing how to implement Xavier and He_normal initializations using TensorFlow:

• Xavier Glorot, Yoshua Bengio: "Understanding the difficulty of training deep feedforward neural networks" • Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification"