Introduction

Sigmoid is one of the classic activation functions in neural networks, especially for binary outputs. When people ask about sigmoid with bias updates, the real issue is how the bias changes the neuron's output and why its gradient is so simple during backpropagation.

Forward Pass

A sigmoid neuron computes sigmoid(w · x + b), where w is the weight vector, x is the input vector, and b is the bias. The bias shifts the activation left or right. Without it, the neuron can only represent boundaries forced through the origin; with it, the neuron can learn useful offsets even when the input is zero. The sigmoid itself is 1 / (1 + exp(-z)), so the output stays between 0 and 1.

Bias matters because it changes the pre-activation value z. If two neurons have the same weights but different biases, they can produce very different probabilities from the same input. That is why the bias is a real trainable parameter, not just a constant decoration on the formula.

Why Bias Updates Are Easy

Let z = w · x + b. The derivative of z with respect to the bias is 1, so once backpropagation gives you dL/dz, you already have the bias gradient: dL/db = dL/dz. The update rule is just gradient descent: b = b - learning_rate * dL/db. Weight updates need an input term because each weight multiplies one feature. The bias does not multiply an input, so its gradient is simpler.

This is a useful mental model: every neuron has one extra trainable degree of freedom that shifts the activation before the nonlinear function is applied. In code, frameworks usually store the bias beside the weights and update both at the same optimization step.

Numerical Example

1import math
2
3x1, x2 = 0.5, 0.3
4w1, w2 = 0.4, 0.7
5b = -0.2
6
7z = w1 * x1 + w2 * x2 + b
8y = 1 / (1 + math.exp(-z))
9
10print(round(z, 4))
11print(round(y, 4))

Output:

0.21
0.5523

Now suppose backpropagation has already produced dL/dz = 0.12 and the learning rate is 0.1.

1learning_rate = 0.1
2dL_dz = 0.12
3b = b - learning_rate * dL_dz
4print(round(b, 4))

Output:

-0.212

That single number 0.12 is also the bias gradient.

Where Sigmoid's Derivative Appears

Sigmoid's derivative is sigmoid(z) * (1 - sigmoid(z)). That factor appears when you differentiate the loss with respect to z. A tiny squared-error example makes the chain rule explicit.

1import math
2
3target = 1.0
4z = 0.21
5prediction = 1 / (1 + math.exp(-z))
6dL_dprediction = prediction - target
7dprediction_dz = prediction * (1 - prediction)
8dL_dz = dL_dprediction * dprediction_dz
9print(round(dL_dz, 6))

Once you compute dL_dz, the bias update is immediate. That is the main reason bias updates are often explained as the simplest part of backpropagation through a neuron.

Why Sigmoid Can Be Difficult

Sigmoid is smooth and interpretable, but it saturates. When z becomes very positive or very negative, the output approaches 1 or 0, and the derivative becomes tiny. In deep hidden layers that creates vanishing gradients and slows learning. Modern networks therefore use ReLU-style activations more often in hidden layers, while sigmoid remains common in the final layer of binary classifiers.

Even when the activation changes, the basic role of the bias stays the same: it shifts the pre-activation value and gets updated from the gradient flowing through that neuron.

Common Pitfalls

• Forgetting that the bias is trainable.
• Thinking the bias gradient needs an input multiplier.
• Mixing up dL/doutput and dL/dz.
• Using sigmoid in many deep hidden layers and then seeing slow learning.
• Treating the bias as optional when it often changes what the neuron can represent.

Summary

• A sigmoid neuron computes sigmoid(w · x + b).
• The bias shifts the activation and improves model flexibility.
• Once backpropagation gives dL/dz, that same value is the bias gradient.
• Bias updates are ordinary gradient-descent steps.
• Sigmoid is still useful, but deep hidden layers often prefer other activations.