Introduction

The num_words argument in Keras Tokenizer often looks broken because it does not shrink word_index after fit_on_texts. That is expected behavior: the tokenizer still records the full vocabulary, and the num_words limit is applied later when texts are converted into sequences or matrices.

What num_words Actually Does

When you call fit_on_texts, Keras counts all observed tokens and assigns indices based on frequency. The internal word_index therefore contains every token the tokenizer saw.

The num_words limit is then used as a cutoff during downstream transformations. In other words:

• vocabulary statistics are full
• generated sequences are truncated to the most common tokens

That is why inspecting word_index alone makes it seem like num_words had no effect.

Demonstration

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = [
4    "red blue blue green",
5    "blue green yellow",
6    "red yellow yellow black",
7]
8
9tokenizer = Tokenizer(num_words=3)
10tokenizer.fit_on_texts(texts)
11
12print(tokenizer.word_index)
13print(tokenizer.texts_to_sequences(["red blue yellow black"]))

You will usually see a word_index with more than three entries, but the sequence output keeps only indices below num_words. Less frequent words are skipped unless you define an out-of-vocabulary token.

Why The Sequence Looks Shorter

Suppose the most frequent tokens receive these indices:

• 'blue becomes 1'
• 'yellow becomes 2'
• 'red becomes 3'
• 'green becomes 4'

With num_words=3, only indices strictly less than 3 are kept during texts_to_sequences. That means tokens at index 3 and above are not included. This catches many people because they expect the top three indexed words to survive, but in practice the cutoff is based on index comparison, not on dictionary length.

Use oov_token When You Need Stable Output

If you want unseen or filtered words to map to a known placeholder instead of disappearing entirely, set oov_token.

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = ["cat sat mat", "cat ate fish", "dog ate fish"]
4
5tokenizer = Tokenizer(num_words=4, oov_token="[OOV]")
6tokenizer.fit_on_texts(texts)
7
8print(tokenizer.word_index)
9print(tokenizer.texts_to_sequences(["cat bird fish turtle"]))

Now filtered or unknown tokens map to the out-of-vocabulary index rather than vanishing. That is often better for model stability.

Match Your Embedding Layer To The Limit

If you use an embedding layer, its input_dim should usually reflect the effective vocabulary limit rather than the raw word_index size.

1from tensorflow.keras import Sequential
2from tensorflow.keras.layers import Embedding, GlobalAveragePooling1D, Dense
3
4model = Sequential([
5    Embedding(input_dim=5000, output_dim=32),
6    GlobalAveragePooling1D(),
7    Dense(1, activation="sigmoid")
8])

If your tokenizer uses num_words=5000, your embedding layer should normally expect that same maximum indexed range, adjusted for padding and any out-of-vocabulary token strategy.

Practical Debugging Checklist

When num_words seems ineffective, check these things in order:

• did you inspect word_index instead of the generated sequences
• did you forget that the cutoff is applied during transformation
• are you using oov_token or silently dropping filtered tokens
• does your embedding layer match the intended vocabulary limit

Most confusion comes from mixing up tokenizer statistics with tokenizer output.

Common Pitfalls

The most common mistake is expecting fit_on_texts to cap word_index. It does not. It records the whole corpus vocabulary.

Another mistake is setting num_words=n and expecting indices up to and including n to remain. The effective sequence filter is based on the tokenizer’s index rules, so borderline indices often surprise people.

A third issue is forgetting oov_token. Without it, rare or filtered words disappear from sequences, which can make debugging harder.

Summary

• 'num_words does not shrink word_index after fitting.'
• The limit is applied when converting texts to sequences or matrices.
• Inspect sequence output, not just the vocabulary dictionary.
• Use oov_token if you want filtered words to map to a known placeholder.
• Keep your embedding layer dimensions aligned with the tokenizer limit.