Introduction

The derivative of ReLU is simple almost everywhere: 1 for positive inputs and 0 for negative inputs. The only subtle point is the value at exactly 0, where ReLU is not differentiable. In NumPy implementations, people usually define the derivative at 0 as 0, which is a practical convention used in many educational implementations and some hand-built neural-network code.

ReLU And Its Derivative

The ReLU activation is:

relu(x) = max(0, x)

Its derivative is commonly implemented as:

• '1 if x > 0,'
• '0 if x <= 0.'

That gives a vectorized NumPy implementation that is easy to write and efficient to run.

A Basic NumPy Implementation

A straightforward implementation uses a boolean comparison and type conversion.

1import numpy as np
2
3
4def relu_derivative(x):
5    x = np.asarray(x)
6    return (x > 0).astype(x.dtype if np.issubdtype(x.dtype, np.floating) else np.float32)
7
8x = np.array([-2.0, -0.5, 0.0, 1.2, 3.4])
9print(relu_derivative(x))

This returns an array of zeros and ones with the same shape as x.

An Even Simpler Variant

Another common implementation uses np.where.

1import numpy as np
2
3
4def relu_derivative_where(x):
5    x = np.asarray(x)
6    return np.where(x > 0, 1.0, 0.0)
7
8print(relu_derivative_where(np.array([-1.0, 0.0, 2.0])))

Both styles are fine. The boolean-cast version is often a little more compact, while np.where can be easier for beginners to read.

What About x == 0?

Mathematically, ReLU is not differentiable at 0. In practical code, you still need a value there.

The most common convention is to set the derivative at 0 to 0.

x = np.array([-1.0, 0.0, 2.0])
print((x > 0).astype(np.float32))

This yields [0., 0., 1.].

Some theoretical discussions choose a subgradient at 0, but for hand-coded NumPy neural-network practice, 0 is the normal choice unless your framework or derivation explicitly says otherwise.

Use The Pre-Activation Input

When implementing backpropagation manually, the derivative should usually be computed from the pre-activation values or from the activation output in a way that preserves the same sign information.

For example, if z is the linear input to ReLU:

z = np.array([[-1.0, 2.0], [0.0, 5.0]])
grad = relu_derivative(z)
print(grad)

This gradient mask is what you multiply elementwise with the upstream gradient.

Applying It In Backpropagation

A simple backpropagation step often looks like this:

1upstream_grad = np.array([[0.2, 0.3], [0.4, 0.5]])
2z = np.array([[-1.0, 2.0], [0.0, 5.0]])
3
4local_grad = relu_derivative(z)
5dz = upstream_grad * local_grad
6
7print(dz)

The entries corresponding to non-positive z values are zeroed out, which matches the behavior of ReLU during backpropagation.

Watch The Dtype

One small but useful detail is output dtype. If your training arrays are float arrays, returning floats for the derivative is usually better than returning booleans.

That is why implementations often cast the mask to float32 or match the floating dtype of the input.

Common Pitfalls

• Forgetting that ReLU is not differentiable exactly at 0 and needing a practical convention.
• Returning booleans when the rest of the computation expects numeric gradient arrays.
• Computing the derivative from the wrong tensor in a manual backpropagation implementation.
• Using Python loops instead of NumPy vectorization.
• Confusing the derivative of ReLU with the ReLU activation itself.

Summary

• The usual NumPy implementation of ReLU derivative is 1 where x > 0 and 0 otherwise.
• At x == 0, most practical implementations choose 0.
• Boolean masks plus casting or np.where both work well.
• In backpropagation, multiply the upstream gradient by the derivative mask elementwise.
• Keep the implementation vectorized and dtype-aware for clean numerical code.