LSTM
Keras
trainable initial state
deep learning
neural networks

How do I make the initial state of an LSTM trainable in Keras?

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Introduction

An LSTM (Long Short-Term Memory) network is a type of recurrent neural network capable of learning long-term dependencies. It is widely used for sequence tasks like time series forecasting, natural language processing, and speech recognition. The internal state of an LSTM comprises two parts: the hidden state hth_t and the cell state ctc_t. By default, Keras initializes both states to zero vectors at the start of each sequence. Making these initial states trainable allows the model to learn an optimal starting point, which can improve performance on tasks where the initial context matters.

Why Make Initial States Trainable?

Zero initialization works well in many cases, but it has limitations:

  • Sequence-to-sequence tasks: When the decoder LSTM needs to "know" something about the input before generating the first output token, a learned initial state can encode that prior knowledge.
  • Short sequences: With very short sequences, the LSTM has few timesteps to build up a useful hidden state. A learned initialization gives it a head start.
  • Stateful models: In some architectures, the initial state carries information from a different part of the model (e.g., an encoder). Making it trainable allows end-to-end optimization.

Implementation in Keras

Keras does not natively support trainable initial states for LSTM layers. You need to create a custom layer that wraps the LSTM and introduces trainable weight variables for the initial hidden and cell states.

Custom Layer Implementation

python
1import tensorflow as tf
2from tensorflow.keras.layers import Layer, LSTM
3
4class TrainableInitialStateLSTM(Layer):
5    def __init__(self, units, **kwargs):
6        super().__init__(**kwargs)
7        self.units = units
8        self.lstm = LSTM(units, return_sequences=True, return_state=True)
9
10    def build(self, input_shape):
11        # Create trainable initial states
12        self.initial_h = self.add_weight(
13            shape=(1, self.units),
14            initializer='zeros',
15            trainable=True,
16            name='initial_h'
17        )
18        self.initial_c = self.add_weight(
19            shape=(1, self.units),
20            initializer='zeros',
21            trainable=True,
22            name='initial_c'
23        )
24        super().build(input_shape)
25
26    def call(self, inputs):
27        batch_size = tf.shape(inputs)[0]
28        # Tile the initial states across the batch dimension
29        h0 = tf.tile(self.initial_h, [batch_size, 1])
30        c0 = tf.tile(self.initial_c, [batch_size, 1])
31        # Pass initial states to the LSTM
32        output, final_h, final_c = self.lstm(
33            inputs, initial_state=[h0, c0]
34        )
35        return output
36
37    def get_config(self):
38        config = super().get_config()
39        config.update({'units': self.units})
40        return config

Using the Custom Layer

python
1from tensorflow.keras.models import Sequential
2from tensorflow.keras.layers import Dense
3
4model = Sequential([
5    TrainableInitialStateLSTM(64, input_shape=(100, 10)),
6    Dense(1)
7])
8
9model.compile(optimizer='adam', loss='mse')
10model.summary()

How It Works

  1. Weight creation: During build(), the layer creates two trainable weight tensors of shape (1,units)(1, \text{units}), one for h0h_0 and one for c0c_0. These start as zeros but are updated by backpropagation during training.
  2. Batch tiling: During call(), the initial states are tiled (replicated) across the batch dimension using tf.tile. This produces tensors of shape (batch_size,units)(\text{batch\_size}, \text{units}), which is the shape the LSTM expects for its initial state.
  3. Gradient flow: Because h0h_0 and c0c_0 are tf.Variable objects registered via add_weight, gradients flow through the LSTM back to these initial states during training. The optimizer updates them just like any other weight in the network.

Alternative Approach: Using a Dense Layer

Another approach encodes the initial state from an external input (like an encoder output) rather than learning a fixed vector:

python
1from tensorflow.keras.layers import Input, Dense, LSTM
2from tensorflow.keras.models import Model
3
4encoder_input = Input(shape=(50, 10))
5encoder_lstm = LSTM(64)(encoder_input)
6
7# Generate initial states for decoder from encoder output
8decoder_h0 = Dense(64, activation='tanh')(encoder_lstm)
9decoder_c0 = Dense(64, activation='tanh')(encoder_lstm)
10
11decoder_input = Input(shape=(100, 10))
12decoder_lstm = LSTM(64, return_sequences=True)
13decoder_output = decoder_lstm(
14    decoder_input, initial_state=[decoder_h0, decoder_c0]
15)
16
17model = Model([encoder_input, decoder_input], decoder_output)

This pattern is common in encoder-decoder architectures for machine translation and summarization.

Practical Considerations

  • Overfitting risk: Adding trainable initial states introduces 2×units2 \times \text{units} extra parameters. For large LSTM sizes, this is negligible, but on very small datasets it could contribute to overfitting. Use dropout or regularization as needed.
  • Initialization strategy: Starting from zeros is standard. Random initialization (e.g., Glorot uniform) is another option but rarely makes a significant difference in practice.
  • Multi-layer LSTMs: If you stack multiple LSTM layers, each layer can have its own trainable initial state. Apply the same wrapper pattern to each layer.
  • Bidirectional LSTMs: For bidirectional models, both the forward and backward LSTMs need separate initial states, doubling the number of trainable initial state parameters to 4×units4 \times \text{units}.

Summary

ApproachWhen to UseComplexity
Zero initialization (default)Most standard tasksNone
Fixed trainable initial stateShort sequences, single-input modelsLow
Dense-projected initial stateEncoder-decoder architecturesMedium

Making LSTM initial states trainable in Keras requires a custom layer wrapper, but the implementation is straightforward. The trainable initial states are tiled across the batch dimension and passed to the LSTM, where gradients flow back to update them during training. This technique is most beneficial for sequence-to-sequence tasks and short-sequence scenarios where the default zero initialization leaves performance on the table.


Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Basis Lab
No course covers this one yet

Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.

Build my course
ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design