How do I make the initial state of an LSTM trainable in Keras?
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
An LSTM (Long Short-Term Memory) network is a type of recurrent neural network capable of learning long-term dependencies. It is widely used for sequence tasks like time series forecasting, natural language processing, and speech recognition. The internal state of an LSTM comprises two parts: the hidden state and the cell state . By default, Keras initializes both states to zero vectors at the start of each sequence. Making these initial states trainable allows the model to learn an optimal starting point, which can improve performance on tasks where the initial context matters.
Why Make Initial States Trainable?
Zero initialization works well in many cases, but it has limitations:
- Sequence-to-sequence tasks: When the decoder LSTM needs to "know" something about the input before generating the first output token, a learned initial state can encode that prior knowledge.
- Short sequences: With very short sequences, the LSTM has few timesteps to build up a useful hidden state. A learned initialization gives it a head start.
- Stateful models: In some architectures, the initial state carries information from a different part of the model (e.g., an encoder). Making it trainable allows end-to-end optimization.
Implementation in Keras
Keras does not natively support trainable initial states for LSTM layers. You need to create a custom layer that wraps the LSTM and introduces trainable weight variables for the initial hidden and cell states.
Custom Layer Implementation
Using the Custom Layer
How It Works
- Weight creation: During
build(), the layer creates two trainable weight tensors of shape , one for and one for . These start as zeros but are updated by backpropagation during training. - Batch tiling: During
call(), the initial states are tiled (replicated) across the batch dimension usingtf.tile. This produces tensors of shape , which is the shape the LSTM expects for its initial state. - Gradient flow: Because and are
tf.Variableobjects registered viaadd_weight, gradients flow through the LSTM back to these initial states during training. The optimizer updates them just like any other weight in the network.
Alternative Approach: Using a Dense Layer
Another approach encodes the initial state from an external input (like an encoder output) rather than learning a fixed vector:
This pattern is common in encoder-decoder architectures for machine translation and summarization.
Practical Considerations
- Overfitting risk: Adding trainable initial states introduces extra parameters. For large LSTM sizes, this is negligible, but on very small datasets it could contribute to overfitting. Use dropout or regularization as needed.
- Initialization strategy: Starting from zeros is standard. Random initialization (e.g., Glorot uniform) is another option but rarely makes a significant difference in practice.
- Multi-layer LSTMs: If you stack multiple LSTM layers, each layer can have its own trainable initial state. Apply the same wrapper pattern to each layer.
- Bidirectional LSTMs: For bidirectional models, both the forward and backward LSTMs need separate initial states, doubling the number of trainable initial state parameters to .
Summary
| Approach | When to Use | Complexity |
| Zero initialization (default) | Most standard tasks | None |
| Fixed trainable initial state | Short sequences, single-input models | Low |
| Dense-projected initial state | Encoder-decoder architectures | Medium |
Making LSTM initial states trainable in Keras requires a custom layer wrapper, but the implementation is straightforward. The trainable initial states are tiled across the batch dimension and passed to the LSTM, where gradients flow back to update them during training. This technique is most beneficial for sequence-to-sequence tasks and short-sequence scenarios where the default zero initialization leaves performance on the table.
Related reading
- How do I pass a scalar via a TensorFlow feed dictionary
- How do I plot a Keras/Tensorflow subclassing API model?
- How do I print the model summary in PyTorch?
- How do I save a trained model in PyTorch?
- How do I pass an OpenCV Mat into a C Tensorflow graph?
- How do I print inside the loss function during training in Keras?
- How do I save and load BatchNormalization Layer in this Tensorflow model?
- How do I set TensorFlow \`RNN\` state when state_is_tupleTrue?
System design interview problems on this topic
- Design an LLM Inference Service: solution and walkthrough
- Design a GPU Cluster Scheduler for ML Training: solution and walkthrough
- Design ChatGPT: solution and walkthrough
- Design an E-commerce Recommendation System: solution and walkthrough
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseNo course covers this one yet
Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.
Build my courseML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.