Introduction

Weight initialization affects how easily a neural network starts learning. In TensorFlow and Keras, you usually do not need to invent an initialization scheme from scratch, but you do need to choose one that matches the layer type and activation function.

Why Initialization Matters

If weights start too small, activations and gradients can shrink toward zero. If they start too large, activations can explode and training becomes unstable. Good initializers try to keep signal magnitude in a reasonable range as data moves forward and gradients move backward.

That is why initialization is tied to layer shape and activation behavior, not just to random numbers.

The Common Built-In Initializers

TensorFlow exposes initializers through tf.keras.initializers. The ones you will see most often are:

• 'GlorotUniform or GlorotNormal, also called Xavier initialization'
• 'HeNormal or HeUniform, often used with ReLU-like activations'
• 'RandomNormal and RandomUniform for manual control'
• 'Zeros and Ones, usually for biases or special cases rather than kernels'

A simple rule of thumb is:

• use Glorot for tanh-like or general dense networks
• use He initialization for ReLU-family activations
• use zeros for biases unless you have a specific reason not to

A Basic Keras Example

Here is a small model that sets initializers explicitly.

1import tensorflow as tf
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Dense(
5        64,
6        activation="relu",
7        kernel_initializer=tf.keras.initializers.HeNormal(),
8        bias_initializer="zeros",
9        input_shape=(20,),
10    ),
11    tf.keras.layers.Dense(
12        32,
13        activation="relu",
14        kernel_initializer=tf.keras.initializers.HeNormal(),
15        bias_initializer="zeros",
16    ),
17    tf.keras.layers.Dense(
18        1,
19        activation="sigmoid",
20        kernel_initializer=tf.keras.initializers.GlorotUniform(),
21        bias_initializer="zeros",
22    ),
23])
24
25model.build()
26model.summary()

The hidden ReLU layers use He initialization because it is designed to preserve variance more effectively for ReLU-like activations. The output layer uses Glorot initialization, which is a sensible general-purpose choice.

Initializers for Standalone Variables

You can also initialize raw TensorFlow variables directly.

1import tensorflow as tf
2
3initializer = tf.keras.initializers.GlorotUniform()
4weights = tf.Variable(initializer(shape=(4, 3)), trainable=True)
5print(weights.shape)

This is useful when writing custom layers or lower-level TensorFlow code outside the standard Dense or Conv layer constructors.

Matching Initializer to Activation

The initializer choice is not arbitrary. It reflects how the activation behaves.

• ReLU drops negative values, so He initialization often works better
• tanh and sigmoid are more symmetric, so Glorot is a common baseline
• very deep or unusual architectures may need custom schemes or empirical tuning

That does not mean other combinations never work. It means the default starting point should be informed by the activation rather than chosen randomly.

What Not to Do

Initializing every weight to zero is a classic mistake. If all neurons in a layer start with the same weights, they receive the same gradients and learn the same features. Randomized initializers break that symmetry.

Biases are different. Zero bias initialization is usually fine because symmetry problems mainly come from identical kernels, not from identical bias values.

Common Pitfalls

The most common mistake is using one initializer everywhere without considering the activation function.

Another mistake is initializing kernels to zeros, which prevents neurons in the same layer from learning distinct patterns.

A third pitfall is treating initialization as a cure-all. Bad learning rates, poor normalization, or unstable architectures cannot always be fixed by changing the initializer.

Summary

• Weight initialization strongly affects optimization stability and learning speed.
• Use TensorFlow's built-in initializers instead of inventing ad hoc random values.
• He initializers are a strong default for ReLU-family layers.
• Glorot initializers are a common general-purpose baseline.
• Keep bias initialization simple and avoid zero-initializing the kernels of trainable layers.