Introduction

In the old standalone CuDNNLSTM layer, the activations were effectively fixed by the cuDNN implementation: tanh for the cell candidate and output state transformation, and sigmoid for the gates. That is why you did not configure custom activation functions there the way you could with the more general LSTM layer.

Why CuDNNLSTM had implicit activations

CuDNNLSTM existed to expose a fast GPU-optimized implementation backed by NVIDIA cuDNN. Speed came partly from specialization, which meant the layer did not support arbitrary combinations of activation functions and recurrent settings.

The common LSTM pieces were effectively fixed as follows:

• input, forget, and output gates used sigmoid
• the cell candidate and output state path used tanh

Those are the classic LSTM activations and exactly what cuDNN expected for its optimized kernel path.

This is different from a fully configurable LSTM

A general Keras LSTM layer historically allowed you to specify options such as:

• 'activation'
• 'recurrent_activation'
• recurrent dropout
• implementation details not compatible with cuDNN fast paths

Example:

1from tensorflow import keras
2from tensorflow.keras import layers
3
4model = keras.Sequential([
5    layers.Input(shape=(20, 8)),
6    layers.LSTM(32, activation="tanh", recurrent_activation="sigmoid"),
7])

With a flexible LSTM, you could choose different activations. With old CuDNNLSTM, you did not get that same freedom because the performance contract was narrower.

The modern Keras answer is usually just LSTM

In current TensorFlow and Keras usage, you normally do not use a separate CuDNNLSTM class anymore. Instead, the regular LSTM layer uses the optimized cuDNN-backed implementation automatically when your configuration is compatible.

That means the old question about the implicit activation function still matters conceptually, because the fast path still expects the usual LSTM activations.

A typical compatible configuration looks like this:

1from tensorflow import keras
2from tensorflow.keras import layers
3
4model = keras.Sequential([
5    layers.Input(shape=(20, 8)),
6    layers.LSTM(
7        32,
8        activation="tanh",
9        recurrent_activation="sigmoid",
10        recurrent_dropout=0.0,
11        use_bias=True,
12    ),
13])

When the layer meets the backend's optimized conditions, Keras can select the GPU-fast implementation under the hood.

Why you should care about the activations

If you change the activations away from tanh and sigmoid, you may lose compatibility with the optimized kernel path and fall back to a slower generic implementation.

That is not automatically bad. Sometimes custom activations are worth it. But you should make the tradeoff consciously rather than assuming every LSTM configuration gets the same GPU acceleration.

The gate behavior still follows standard LSTM math

The reason sigmoid is used for gates is that gates need values in a bounded control range, effectively between "mostly closed" and "mostly open." The reason tanh is used for the candidate and hidden-state transformation is that it keeps those values centered and bounded.

So the implicit activations were not arbitrary. They reflect the conventional LSTM formulation that cuDNN optimized for.

If you need custom recurrent behavior, use the general layer

If your model depends on nonstandard activations or other settings outside the optimized path, use the normal LSTM layer and accept that the runtime may not use the fastest cuDNN kernel.

That is usually a better design than trying to force a specialized kernel abstraction to do something it was not built to support.

Common Pitfalls

• Assuming old CuDNNLSTM supported arbitrary activation choices like the general LSTM layer.
• Forgetting that the fast GPU path depends on compatible LSTM settings.
• Treating a fallback to the generic implementation as a bug rather than a configuration consequence.
• Copying legacy CuDNNLSTM examples into modern codebases that should just use LSTM.
• Changing activations without realizing that the performance characteristics may change too.

Summary

• In legacy CuDNNLSTM, the effective activations were fixed to the standard LSTM choices: tanh and sigmoid.
• That restriction existed because the layer was specialized for cuDNN performance.
• Modern Keras usually uses LSTM and selects the cuDNN-optimized path automatically when settings are compatible.
• Custom activations may disable the fast path.
• Use the general LSTM layer when flexibility matters more than strict compatibility with the optimized kernel.