LSTM
TensorFlow
word prediction
language model
machine learning

Predicting the next word using the LSTM ptb model tensorflow example

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Predicting the next word in a sequence is a core task in Natural Language Processing (NLP) and the foundation of autocomplete, text generation, and conversational AI systems. The Long Short-Term Memory (LSTM) network has been one of the most effective architectures for this task. This article walks through building an LSTM-based language model using TensorFlow, trained on the Penn Treebank (PTB) dataset, with code examples and explanations of the key concepts.

Why LSTMs for Word Prediction

Traditional recurrent neural networks (RNNs) struggle with long-term dependencies because gradients either vanish or explode during backpropagation through time. LSTMs solve this problem with a gating mechanism that controls what information to remember, forget, and output at each time step.

Key Components of an LSTM Cell

  • Cell state: A separate memory channel that runs through the entire sequence. It carries relevant information forward with minimal transformation, which is what allows LSTMs to remember patterns across long spans.
  • Forget gate: Decides what to discard from the cell state based on the current input and previous hidden state.
  • Input gate: Determines which new values from the current input should be written into the cell state.
  • Output gate: Controls which parts of the cell state to expose as the hidden state output for the current time step.

The Penn Treebank Dataset

The PTB dataset is a standard benchmark for language modeling. It consists of pre-tokenized Wall Street Journal articles from 1989, with a vocabulary of about 10,000 unique words. The dataset is split into training, validation, and test sets. Its small vocabulary makes it practical for experimentation while still being a meaningful test of model quality.

Building the LSTM Language Model

Step 1: Prepare the Data

Load the PTB data and convert words to integer indices using a vocabulary mapping.

python
1import numpy as np
2import tensorflow as tf
3
4def load_ptb_data(file_path):
5    with open(file_path, 'r') as f:
6        words = f.read().replace('\n', '<eos>').split()
7    return words
8
9def build_vocab(words):
10    word_counts = {}
11    for w in words:
12        word_counts[w] = word_counts.get(w, 0) + 1
13    sorted_words = sorted(word_counts, key=word_counts.get, reverse=True)
14    word_to_id = {w: i for i, w in enumerate(sorted_words)}
15    return word_to_id
16
17def encode(words, word_to_id):
18    return np.array([word_to_id[w] for w in words if w in word_to_id])
19
20words = load_ptb_data('ptb.train.txt')
21word_to_id = build_vocab(words)
22train_data = encode(words, word_to_id)
23vocab_size = len(word_to_id)

Step 2: Create Training Sequences

Language models learn to predict the next word given a context window. You need to create input-target pairs where the target is the input shifted by one position.

python
1def create_sequences(data, seq_length):
2    inputs, targets = [], []
3    for i in range(0, len(data) - seq_length):
4        inputs.append(data[i:i + seq_length])
5        targets.append(data[i + 1:i + seq_length + 1])
6    return np.array(inputs), np.array(targets)
7
8SEQ_LENGTH = 35
9X_train, y_train = create_sequences(train_data, SEQ_LENGTH)

Each input sequence has a corresponding target sequence that is shifted by one time step. This lets the model learn to predict the next token at every position.

Step 3: Define the LSTM Model

python
1EMBEDDING_DIM = 200
2HIDDEN_DIM = 200
3
4model = tf.keras.Sequential([
5    tf.keras.layers.Embedding(vocab_size, EMBEDDING_DIM, input_length=SEQ_LENGTH),
6    tf.keras.layers.LSTM(HIDDEN_DIM, return_sequences=True),
7    tf.keras.layers.Dropout(0.5),
8    tf.keras.layers.LSTM(HIDDEN_DIM, return_sequences=True),
9    tf.keras.layers.Dropout(0.5),
10    tf.keras.layers.Dense(vocab_size, activation='softmax')
11])
12
13model.compile(
14    loss='sparse_categorical_crossentropy',
15    optimizer='adam',
16    metrics=['accuracy']
17)
18
19model.summary()

The embedding layer converts word indices into dense vectors. Two stacked LSTM layers with dropout provide the sequence modeling capacity. The dense output layer produces a probability distribution over the entire vocabulary for each time step.

Step 4: Train the Model

python
1model.fit(
2    X_train, y_train,
3    epochs=10,
4    batch_size=64,
5    validation_split=0.1
6)

Step 5: Predict the Next Word

Once trained, you can feed a sequence of words and get the model's prediction for what comes next.

python
1def predict_next_word(model, input_text, word_to_id, id_to_word, seq_length):
2    tokens = input_text.lower().split()
3    encoded = [word_to_id.get(w, 0) for w in tokens[-seq_length:]]
4    padded = tf.keras.preprocessing.sequence.pad_sequences(
5        [encoded], maxlen=seq_length, padding='pre'
6    )
7    prediction = model.predict(padded, verbose=0)
8    predicted_id = np.argmax(prediction[0, -1, :])
9    return id_to_word[predicted_id]
10
11id_to_word = {i: w for w, i in word_to_id.items()}
12next_word = predict_next_word(model, "the company said it", word_to_id, id_to_word, SEQ_LENGTH)
13print(f"Predicted next word: {next_word}")

The function encodes the input text, pads it to the expected sequence length, runs it through the model, and selects the word with the highest probability from the output distribution.

Evaluating with Perplexity

Language models are typically evaluated using perplexity, which measures how surprised the model is by the test data. Lower perplexity means better prediction quality.

python
loss = model.evaluate(X_test, y_test, verbose=0)
perplexity = np.exp(loss[0])
print(f"Test perplexity: {perplexity:.2f}")

A well-trained single-layer LSTM on PTB typically achieves a perplexity between 80 and 100. Stacking layers and tuning hyperparameters can push this lower.

Common Pitfalls

  • Vocabulary mismatch: If a word at inference time is not in the training vocabulary, the model cannot process it. Use an unknown token (often <unk>) as a fallback.
  • Sequence length trade-off: Longer sequences give the model more context but require more memory and computation. The PTB example commonly uses a sequence length of 20 to 35.
  • Overfitting on small data: The PTB dataset is small by modern standards. Use dropout, weight tying (sharing the embedding and output layer weights), and learning rate decay to regularize.
  • Gradient clipping: LSTMs can still experience gradient issues during training. Clip gradients to a maximum norm (typically 5.0) to stabilize training.

Summary

Building a next-word prediction model with LSTMs and TensorFlow involves preparing tokenized data, creating shifted input-target pairs, and training a stacked LSTM architecture with embedding and dense output layers. The PTB dataset provides a practical benchmark for evaluating language models, with perplexity as the standard metric. While modern transformer architectures have largely surpassed LSTMs in production systems, understanding LSTM-based language modeling provides essential foundations for working with sequence prediction tasks.


Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Basis Lab
There is a whole course on this

Basis Lab has written “How a Model Picks Its Next Word”, a course that teaches this from the ground up rather than answering the single question. It is a separate product from the same team and the course is free to read.

Read the course
ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design