Introduction
Perplexity measures how well a language model predicts a sequence of words. A lower perplexity means the model is less "surprised" by the test data and assigns higher probabilities to the actual next words. For an RNN language model in TensorFlow, perplexity is calculated as the exponential of the average cross-entropy loss per token. The formula is perplexity = exp(average_loss), where the loss comes from tf.nn.sparse_softmax_cross_entropy_with_logits.
Perplexity is defined as:
PP(W) = exp( -1/N * sum(log P(w_i | w_1, ..., w_{i-1})) )
Where N is the total number of tokens and P(w_i | context) is the probability the model assigns to the actual next word. In practice, this simplifies to:
perplexity = exp(cross_entropy_loss)
Because cross-entropy loss is already the negative log probability averaged over tokens.
Computing Perplexity in TensorFlow
1import tensorflow as tf
2import numpy as np
3
4# Assume an RNN language model that outputs logits
5# logits shape: (batch_size, sequence_length, vocab_size)
6# labels shape: (batch_size, sequence_length)
7
8# Step 1: Compute cross-entropy loss per token
9loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
10 labels=labels, # (batch, seq_len) — integer token IDs
11 logits=logits # (batch, seq_len, vocab_size) — raw scores
12)
13# loss shape: (batch_size, sequence_length)
14
15# Step 2: Mask out padding tokens
16# mask is 1.0 for real tokens, 0.0 for padding
17mask = tf.cast(tf.not_equal(labels, pad_token_id), tf.float32)
18
19# Step 3: Average loss over real tokens only
20masked_loss = loss * mask
21average_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(mask)
22
23# Step 4: Perplexity = exp(average_loss)
24perplexity = tf.exp(average_loss)
Full RNN Language Model Example
1import tensorflow as tf
2
3class RNNLanguageModel(tf.keras.Model):
4 def __init__(self, vocab_size, embed_dim, hidden_dim):
5 super().__init__()
6 self.embedding = tf.keras.layers.Embedding(vocab_size, embed_dim)
7 self.rnn = tf.keras.layers.LSTM(hidden_dim, return_sequences=True)
8 self.output_layer = tf.keras.layers.Dense(vocab_size)
9
10 def call(self, inputs):
11 x = self.embedding(inputs)
12 x = self.rnn(x)
13 logits = self.output_layer(x)
14 return logits
15
16# Build model
17vocab_size = 10000
18model = RNNLanguageModel(vocab_size, embed_dim=256, hidden_dim=512)
19
20# Training step
21optimizer = tf.keras.optimizers.Adam(1e-3)
22
23@tf.function
24def train_step(inputs, targets, mask):
25 with tf.GradientTape() as tape:
26 logits = model(inputs)
27 loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
28 labels=targets, logits=logits
29 )
30 masked_loss = tf.reduce_sum(loss * mask) / tf.reduce_sum(mask)
31
32 gradients = tape.gradient(masked_loss, model.trainable_variables)
33 optimizer.apply_gradients(zip(gradients, model.trainable_variables))
34 return masked_loss
35
36# Evaluation — compute perplexity over entire dataset
37def compute_perplexity(model, dataset, pad_token_id=0):
38 total_loss = 0.0
39 total_tokens = 0
40
41 for inputs, targets in dataset:
42 logits = model(inputs, training=False)
43 loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
44 labels=targets, logits=logits
45 )
46 mask = tf.cast(tf.not_equal(targets, pad_token_id), tf.float32)
47 total_loss += tf.reduce_sum(loss * mask).numpy()
48 total_tokens += tf.reduce_sum(mask).numpy()
49
50 avg_loss = total_loss / total_tokens
51 return np.exp(avg_loss)
52
53# After training:
54ppl = compute_perplexity(model, test_dataset)
55print(f"Test perplexity: {ppl:.2f}")
56# Good language model: 50-100
57# State-of-the-art: 15-30
Perplexity with Keras Built-in Loss
If you use Keras model.fit(), extract perplexity from the reported loss:
1model.compile(
2 optimizer='adam',
3 loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
4 metrics=['accuracy']
5)
6
7history = model.fit(train_dataset, epochs=10, validation_data=val_dataset)
8
9# Perplexity from validation loss
10val_loss = history.history['val_loss'][-1]
11val_perplexity = np.exp(val_loss)
12print(f"Validation perplexity: {val_perplexity:.2f}")
This works only if all tokens are real (no padding). With padding, use a custom metric or sample_weight to mask padded positions.
Interpreting Perplexity Values
| Perplexity | Interpretation |
| 1.0 | Perfect prediction (model is certain of every token) |
| 10-30 | Excellent (state-of-the-art transformer models) |
| 50-100 | Good (well-trained RNN on moderate vocabulary) |
| 100-300 | Fair (baseline or undertrained model) |
| > vocab_size | Worse than uniform random guessing |
Common Pitfalls
Not masking padding tokens: If your sequences are padded, the loss on padding tokens artificially lowers perplexity. Always mask padding before averaging the loss.
Using tf.reduce_mean instead of masked mean: tf.reduce_mean(loss) divides by total positions including padding. Use sum(loss * mask) / sum(mask) for correct per-token average.
Confusing base-e with base-2: exp(loss) gives standard perplexity (base-e). Some papers report base-2 perplexity using 2^loss. Make sure you know which convention is being used.
Computing perplexity during training (not eval): Dropout and other regularization layers change behavior between training and inference. Always compute perplexity with training=False or model.eval().
Numerical overflow for very high loss: If the model is untrained, exp(loss) can overflow. Cap the loss or use tf.clip_by_value before computing exp.
Summary
Perplexity = exp(average cross-entropy loss per token) — lower is better
Use tf.nn.sparse_softmax_cross_entropy_with_logits to compute per-token loss
Always mask padding tokens before averaging the loss
Compute perplexity on validation/test data with dropout disabled
Good RNN perplexity is typically 50-100; state-of-the-art transformers achieve 15-30
