Introduction

An Echo State Network, or ESN, is a recurrent model with a fixed reservoir and a trainable readout. In TensorFlow, that means the custom recurrent logic should update hidden state over time while keeping the input and recurrent reservoir weights non-trainable.

What Makes an ESN Different from a Usual RNN

A standard RNN learns recurrent weights with backpropagation through time. An ESN deliberately does not. The reservoir is randomly initialized, scaled so its dynamics remain useful, and then left fixed while only the final readout is trained.

That changes the implementation goals:

• the reservoir weights are created once and frozen
• the sequence still updates state recurrently at each time step
• the output layer is the part that learns from data

So the main question is not "how do I train a custom RNN end to end?" It is "how do I express a fixed recurrent reservoir inside TensorFlow cleanly?"

A Practical Custom ESN Cell

The Keras RNN wrapper can run any cell-like layer that exposes state_size and output_size. That makes a custom ESN cell a good fit.

1import numpy as np
2import tensorflow as tf
3
4
5class ESNCell(tf.keras.layers.Layer):
6    def __init__(self, units, spectral_radius=0.9, leak_rate=1.0, **kwargs):
7        super().__init__(**kwargs)
8        self.units = units
9        self.spectral_radius = spectral_radius
10        self.leak_rate = leak_rate
11        self.state_size = units
12        self.output_size = units
13
14    def build(self, input_shape):
15        input_dim = int(input_shape[-1])
16
17        w_in = np.random.normal(scale=0.1, size=(input_dim, self.units)).astype("float32")
18        w_res = np.random.normal(scale=0.5, size=(self.units, self.units)).astype("float32")
19
20        eigenvalues = np.linalg.eigvals(w_res)
21        radius = np.max(np.abs(eigenvalues))
22        w_res = w_res * (self.spectral_radius / radius)
23
24        self.w_in = self.add_weight(
25            name="w_in",
26            shape=(input_dim, self.units),
27            initializer=tf.keras.initializers.Constant(w_in),
28            trainable=False,
29        )
30        self.w_res = self.add_weight(
31            name="w_res",
32            shape=(self.units, self.units),
33            initializer=tf.keras.initializers.Constant(w_res),
34            trainable=False,
35        )
36        self.bias = self.add_weight(
37            name="bias",
38            shape=(self.units,),
39            initializer="zeros",
40            trainable=False,
41        )
42
43    def call(self, inputs, states):
44        prev_state = states[0]
45        candidate = tf.tanh(
46            tf.matmul(inputs, self.w_in) +
47            tf.matmul(prev_state, self.w_res) +
48            self.bias
49        )
50        next_state = (1.0 - self.leak_rate) * prev_state + self.leak_rate * candidate
51        return next_state, [next_state]

This cell has the key ESN property: the reservoir weights are part of the layer, but they are not trainable.

Wrap the Cell in a Keras Model

Once the cell exists, you can use the normal tf.keras.layers.RNN wrapper and add a trainable readout layer on top.

1import tensorflow as tf
2
3inputs = tf.keras.Input(shape=(20, 4))
4x = tf.keras.layers.RNN(ESNCell(units=128), return_sequences=False)(inputs)
5outputs = tf.keras.layers.Dense(1)(x)
6
7model = tf.keras.Model(inputs=inputs, outputs=outputs)
8model.compile(optimizer="adam", loss="mse")
9
10dummy_x = tf.random.normal((8, 20, 4))
11dummy_y = tf.random.normal((8, 1))
12model.fit(dummy_x, dummy_y, epochs=1)

In this model, the Dense layer is the trainable readout. The ESN reservoir transforms the sequence into a state representation, and the readout learns how to map that state to the target.

Why Spectral Radius Matters

ESNs are sensitive to reservoir dynamics. If the recurrent matrix is too small, the state may wash out too quickly and stop carrying useful temporal information. If it is too large, the dynamics can become unstable.

That is why many ESN implementations scale the recurrent matrix by spectral radius. The code above does that during build. It is a simplified version, but it captures the standard idea: generate random weights, compute the current radius, then rescale the matrix to the radius you want.

In real experiments, reservoir size, sparsity, input scaling, and leak rate often need to be tuned together. The TensorFlow implementation is only one part of getting a good ESN.

Add Washout When Training

Many ESN workflows ignore the earliest states in each sequence because they are overly influenced by the initial zero state. This is usually called washout.

One simple way to expose later states is to return the full state sequence and slice off the first part before the readout.

1inputs = tf.keras.Input(shape=(20, 4))
2states = tf.keras.layers.RNN(ESNCell(units=64), return_sequences=True)(inputs)
3washed = states[:, 5:, :]
4pooled = tf.reduce_mean(washed, axis=1)
5outputs = tf.keras.layers.Dense(1)(pooled)
6
7model = tf.keras.Model(inputs=inputs, outputs=outputs)

This is not the only washout strategy, but it shows the idea clearly: do not force the readout to learn from unstable early reservoir states.

A Custom Layer Is Also Acceptable

You do not have to implement an ESN as an RNN cell. A custom layer that loops over the time axis can also work. The cell-based approach is simply more aligned with Keras conventions and integrates better with masking, return_sequences, and other recurrent-model features.

That said, the real ESN requirement is architectural, not inheritance-based. If the reservoir is fixed and only the readout learns, the design is still ESN-like even if you do not subclass a formal RNN cell type.

Common Pitfalls

One common mistake is leaving reservoir weights trainable. Once the recurrent or input reservoir weights start training normally, the model is no longer behaving like a classic ESN.

Another pitfall is ignoring spectral radius. Random recurrent weights without scaling can produce reservoir dynamics that are too weak or too unstable.

A third issue is skipping washout entirely and training on states dominated by initialization artifacts.

Finally, do not assume TensorFlow has a built-in ESN layer. You usually need to express the reservoir behavior yourself with a custom cell or layer.

Summary

• An ESN keeps the reservoir fixed and trains only the readout layer.
• In TensorFlow, a custom cell wrapped by tf.keras.layers.RNN is a clean implementation strategy.
• Mark the reservoir weights as non-trainable so the model keeps ESN behavior.
• Scale the recurrent matrix to a sensible spectral radius and consider washout during training.
• The important part is the fixed-reservoir design, not a specific inheritance hierarchy.