Introduction

In a neural network layer, the bias term shifts the pre-activation value so the layer is not forced to pass through the origin. The proper implementation is usually simple: store one learnable bias per output unit, add it after the weighted sum, and let backpropagation update it with the rest of the parameters.

What the Bias Does

For a dense layer, the standard computation is:

output = activation(input * weights + bias)

Without the bias, the layer can only learn transformations constrained by the weighted input sum. The bias gives the model an extra degree of freedom by shifting that sum before the activation function is applied.

Bias Shape Matters

For a dense layer with input_dim inputs and output_dim outputs:

• weights usually have shape input_dim x output_dim
• bias usually has shape output_dim

There is one bias value per output unit, not one bias per input feature.

Simple NumPy Example

1import numpy as np
2
3x = np.array([[1.0, 2.0]])
4W = np.array([
5    [0.5, -1.0],
6    [1.5,  0.2],
7])
8b = np.array([0.1, -0.3])
9
10z = x @ W + b
11print(z)

The bias vector broadcasts across the batch dimension. That is the standard behavior you want in most frameworks.

Biases in a Custom Neural Layer

If you are implementing a layer yourself, keep the bias separate from the weight matrix rather than hiding it inside the input vector unless you have a specific mathematical reason to do so.

1import numpy as np
2
3
4class DenseLayer:
5    def __init__(self, in_features, out_features):
6        self.W = np.random.randn(in_features, out_features) * 0.1
7        self.b = np.zeros(out_features)
8
9    def forward(self, x):
10        return x @ self.W + self.b
11
12
13layer = DenseLayer(3, 2)
14x = np.array([[1.0, 2.0, 3.0]])
15print(layer.forward(x))

Initializing biases to zero is common and usually fine, because symmetry problems in dense layers come from weight initialization, not from zero biases.

Framework Example in Keras

Most high-level frameworks handle bias parameters automatically unless you disable them.

1from tensorflow import keras
2
3layer = keras.layers.Dense(
4    units=4,
5    use_bias=True,
6    bias_initializer="zeros"
7)
8
9model = keras.Sequential([
10    keras.layers.Input(shape=(3,)),
11    layer
12])
13
14print(layer.get_config()["use_bias"])

This is the normal and correct setup for a standard dense layer.

When Biases Are Often Disabled

There are valid cases where you intentionally omit bias terms:

• a linear layer immediately followed by batch normalization
• architectures where a later normalization layer already introduces a learned offset
• certain constrained or analytical models

For example, if batch normalization comes directly after a dense layer, the learned shift from normalization can make the preceding bias redundant. That is an optimization choice, not a general rule to remove biases everywhere.

Convolution Layers Follow the Same Principle

Convolution layers also usually use one bias per output channel, not one per spatial location.

Conceptually:

• kernel weights learn spatial patterns
• bias shifts the whole output channel

That keeps the parameterization efficient and consistent with how feature maps are produced.

Training Behavior

Biases are learned the same way as weights: gradients are computed and the optimizer updates them. The gradient for a bias is usually the accumulated contribution of the output error across the batch for that unit or channel.

You normally do not need special optimizer logic for biases unless you are designing a custom training rule.

Common Pitfalls

The biggest mistake is assigning one bias per input feature instead of one bias per output unit. That does not match how standard neural layers are defined.

Another issue is folding the bias into the weight matrix too early in code meant for clarity or education. That can obscure shapes and make debugging harder.

A third problem is keeping biases enabled in layers where a following normalization layer already provides the same kind of shift and makes the extra parameter unnecessary.

Summary

• Use one learnable bias per output unit or output channel.
• Add the bias after the weighted sum and before or as part of the activation path.
• Zero initialization is usually a reasonable default for bias vectors.
• Frameworks such as Keras already implement biases correctly when use_bias=True.
• Disable bias only when the surrounding architecture makes it redundant.