Introduction

tanh was historically a popular hidden-layer activation in multilayer perceptrons because it is nonlinear, smooth, and zero-centered. It is no longer the default answer for every dense network, but understanding why people used it explains both classic neural-network practice and the situations where tanh is still a sensible choice.

What tanh Does

The hyperbolic tangent maps any real input to the range -1 to 1.

1import numpy as np
2
3x = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
4y = np.tanh(x)
5print(y)

Output values are negative for negative inputs, positive for positive inputs, and close to zero around the origin.

That matters because an activation function is not just about adding nonlinearity. It also shapes the distribution of signals and gradients flowing through the network.

Why tanh Was Preferred Over Sigmoid in Many MLPs

A logistic sigmoid outputs values between 0 and 1. That works, but the activations are not centered around zero. tanh is zero-centered, which often makes optimization easier because positive and negative activations can balance each other more naturally.

In practice, that gave tanh a few important advantages over sigmoid in hidden layers:

• zero-centered activations
• stronger gradients near zero
• more symmetric signal flow
• often faster training in classic MLP settings

A small comparison:

1import numpy as np
2
3x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])
4print("sigmoid-like range is 0 to 1")
5print("tanh:", np.tanh(x))

The exact numbers are less important than the shape: tanh(0) is 0, not 0.5.

Why Zero-Centered Outputs Help

When hidden activations are centered around zero, gradient-based updates can behave more cleanly. If every activation is positive, parameter updates may become more correlated in direction, which can slow optimization.

This is one reason older neural-network textbooks and tutorials often recommended:

• sigmoid for output probabilities in binary classification
• 'tanh for hidden layers in shallow MLPs'

That advice was not arbitrary. It reflected real training behavior before ReLU-style activations became dominant.

The Main Weakness: Saturation

tanh is not a perfect activation. For very large positive or negative inputs, it saturates near 1 or -1. In those regions, the gradient becomes very small.

That means deep networks can still suffer from vanishing gradients with tanh, especially if initialization and normalization are poor.

A quick illustration:

1import numpy as np
2
3x = np.array([-10.0, -3.0, 0.0, 3.0, 10.0])
4print(np.tanh(x))

The outputs near the extremes barely change. That flattening is the reason tanh becomes harder to optimize in deeper networks compared with activations like ReLU.

Why ReLU Often Replaced It

Modern deep learning often prefers ReLU and related activations because they are simpler and reduce some vanishing-gradient issues in deep stacks.

So the modern answer to "Why use tanh?" is not "because it is always best." It is closer to this:

• 'tanh was better than sigmoid for many hidden-layer MLPs'
• ReLU is often better for deeper modern networks
• 'tanh still has useful properties in some settings'

Those useful settings include:

• small or medium fully connected networks
• recurrent models or gates that want bounded signals
• architectures where negative activations are meaningful
• problems where a centered bounded activation helps stabilize behavior

A Small Keras Example

Using tanh in a dense MLP is straightforward:

1from tensorflow import keras
2
3model = keras.Sequential([
4    keras.layers.Input(shape=(20,)),
5    keras.layers.Dense(32, activation="tanh"),
6    keras.layers.Dense(16, activation="tanh"),
7    keras.layers.Dense(1, activation="sigmoid")
8])
9
10model.compile(optimizer="adam", loss="binary_crossentropy")
11model.summary()

This is still a perfectly reasonable architecture for some tabular or low-dimensional problems.

When tanh Is a Good Choice Today

You might choose tanh when you want hidden states that stay bounded and symmetric around zero. That can be desirable when the next layer expects signals with both signs represented naturally.

You might avoid it when you are building a deep feedforward network where training speed and gradient flow are the main priorities. In that case, ReLU-family activations are usually the first thing to try.

So tanh is neither obsolete nor universal. It is a tradeoff.

Common Pitfalls

The biggest mistake is repeating the old rule "always use tanh in MLPs" without context. That advice came from an era when sigmoid was the main comparison point, not from a world where ReLU dominates deep dense networks.

Another common issue is ignoring input scaling. If the activations are driven too far into saturation, tanh loses its training advantage because gradients shrink near the extremes.

Developers also confuse the best hidden-layer activation with the best output-layer activation. tanh may be helpful in hidden layers, but binary classification outputs still often want a sigmoid probability head.

Finally, do not evaluate activations in isolation. Initialization, normalization, optimizer choice, and network depth all affect whether tanh performs well.

Summary

• 'tanh is nonlinear, smooth, bounded, and zero-centered.'
• It was often preferred over sigmoid for hidden layers in classic MLPs.
• Its main weakness is saturation, which can lead to vanishing gradients.
• ReLU is often preferred in deeper modern networks.
• 'tanh is still useful when bounded, symmetric hidden activations are desirable.'