How to calculate perplexity of RNN in tensorflow

RNN

TensorFlow

Perplexity

Machine Learning

Natural Language Processing

How to calculate perplexity of RNN in tensorflow

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Deep Learning TensorFlow & Keras Natural Language Processing Machine Learning

Introduction

Perplexity measures how well a language model predicts a sequence of words. A lower perplexity means the model is less "surprised" by the test data and assigns higher probabilities to the actual next words. For an RNN language model in TensorFlow, perplexity is calculated as the exponential of the average cross-entropy loss per token. The formula is perplexity = exp(average_loss), where the loss comes from tf.nn.sparse_softmax_cross_entropy_with_logits.

The Formula

Perplexity is defined as:

PP(W) = exp( -1/N * sum(log P(w_i | w_1, ..., w_{i-1})) )

Where N is the total number of tokens and P(w_i | context) is the probability the model assigns to the actual next word. In practice, this simplifies to:

perplexity = exp(cross_entropy_loss)

Because cross-entropy loss is already the negative log probability averaged over tokens.

Computing Perplexity in TensorFlow

python

1import tensorflow as tf
2import numpy as np
3
4# Assume an RNN language model that outputs logits
5# logits shape: (batch_size, sequence_length, vocab_size)
6# labels shape: (batch_size, sequence_length)
7
8# Step 1: Compute cross-entropy loss per token
9loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
10    labels=labels,   # (batch, seq_len) — integer token IDs
11    logits=logits     # (batch, seq_len, vocab_size) — raw scores
12)
13# loss shape: (batch_size, sequence_length)
14
15# Step 2: Mask out padding tokens
16# mask is 1.0 for real tokens, 0.0 for padding
17mask = tf.cast(tf.not_equal(labels, pad_token_id), tf.float32)
18
19# Step 3: Average loss over real tokens only
20masked_loss = loss * mask
21average_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(mask)
22
23# Step 4: Perplexity = exp(average_loss)
24perplexity = tf.exp(average_loss)

Full RNN Language Model Example

python

1import tensorflow as tf
2
3class RNNLanguageModel(tf.keras.Model):
4    def __init__(self, vocab_size, embed_dim, hidden_dim):
5        super().__init__()
6        self.embedding = tf.keras.layers.Embedding(vocab_size, embed_dim)
7        self.rnn = tf.keras.layers.LSTM(hidden_dim, return_sequences=True)
8        self.output_layer = tf.keras.layers.Dense(vocab_size)
9
10    def call(self, inputs):
11        x = self.embedding(inputs)
12        x = self.rnn(x)
13        logits = self.output_layer(x)
14        return logits
15
16# Build model
17vocab_size = 10000
18model = RNNLanguageModel(vocab_size, embed_dim=256, hidden_dim=512)
19
20# Training step
21optimizer = tf.keras.optimizers.Adam(1e-3)
22
23@tf.function
24def train_step(inputs, targets, mask):
25    with tf.GradientTape() as tape:
26        logits = model(inputs)
27        loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
28            labels=targets, logits=logits
29        )
30        masked_loss = tf.reduce_sum(loss * mask) / tf.reduce_sum(mask)
31
32    gradients = tape.gradient(masked_loss, model.trainable_variables)
33    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
34    return masked_loss
35
36# Evaluation — compute perplexity over entire dataset
37def compute_perplexity(model, dataset, pad_token_id=0):
38    total_loss = 0.0
39    total_tokens = 0
40
41    for inputs, targets in dataset:
42        logits = model(inputs, training=False)
43        loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
44            labels=targets, logits=logits
45        )
46        mask = tf.cast(tf.not_equal(targets, pad_token_id), tf.float32)
47        total_loss += tf.reduce_sum(loss * mask).numpy()
48        total_tokens += tf.reduce_sum(mask).numpy()
49
50    avg_loss = total_loss / total_tokens
51    return np.exp(avg_loss)
52
53# After training:
54ppl = compute_perplexity(model, test_dataset)
55print(f"Test perplexity: {ppl:.2f}")
56# Good language model: 50-100
57# State-of-the-art: 15-30

Perplexity with Keras Built-in Loss

If you use Keras model.fit(), extract perplexity from the reported loss:

python

1model.compile(
2    optimizer='adam',
3    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
4    metrics=['accuracy']
5)
6
7history = model.fit(train_dataset, epochs=10, validation_data=val_dataset)
8
9# Perplexity from validation loss
10val_loss = history.history['val_loss'][-1]
11val_perplexity = np.exp(val_loss)
12print(f"Validation perplexity: {val_perplexity:.2f}")

This works only if all tokens are real (no padding). With padding, use a custom metric or sample_weight to mask padded positions.

Interpreting Perplexity Values

Perplexity	Interpretation
1.0	Perfect prediction (model is certain of every token)
10-30	Excellent (state-of-the-art transformer models)
50-100	Good (well-trained RNN on moderate vocabulary)
100-300	Fair (baseline or undertrained model)
> vocab_size	Worse than uniform random guessing

Common Pitfalls

Not masking padding tokens: If your sequences are padded, the loss on padding tokens artificially lowers perplexity. Always mask padding before averaging the loss.
Using tf.reduce_mean instead of masked mean: tf.reduce_mean(loss) divides by total positions including padding. Use sum(loss * mask) / sum(mask) for correct per-token average.
Confusing base-e with base-2: exp(loss) gives standard perplexity (base-e). Some papers report base-2 perplexity using 2^loss. Make sure you know which convention is being used.
Computing perplexity during training (not eval): Dropout and other regularization layers change behavior between training and inference. Always compute perplexity with training=False or model.eval().
Numerical overflow for very high loss: If the model is untrained, exp(loss) can overflow. Cap the loss or use tf.clip_by_value before computing exp.

Summary

Perplexity = exp(average cross-entropy loss per token) — lower is better
Use tf.nn.sparse_softmax_cross_entropy_with_logits to compute per-token loss
Always mask padding tokens before averaging the loss
Compute perplexity on validation/test data with dropout disabled
Good RNN perplexity is typically 50-100; state-of-the-art transformers achieve 15-30

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

How to calculate perplexity of RNN in tensorflow

ML System Design practice on Codemia

Introduction

The Formula

Computing Perplexity in TensorFlow

Full RNN Language Model Example

Perplexity with Keras Built-in Loss

Interpreting Perplexity Values

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

Track what you have practised

ML System Design practice on Codemia