Introduction

In Keras text preprocessing, Tokenizer.num_words is a vocabulary cutoff, not a promise that the tokenizer will forget every other word exists. The subtle but important detail is that the tokenizer still counts all words when fitting. The num_words limit is applied later when texts are converted into sequences or matrices.

What num_words Means

The tokenizer ranks words by frequency. The most common word gets the smallest index, the next-most common word gets the next index, and so on.

When you set num_words, you are saying:

• keep building statistics from the full corpus
• but only use the top-ranked words when turning text into model-ready numeric output

That means num_words controls the effective usable vocabulary during vectorization, not the entire internal bookkeeping state.

A Small Example

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = [
4    "apple banana apple orange",
5    "banana apple pear",
6    "pear grape apple",
7]
8
9tokenizer = Tokenizer(num_words=3)
10tokenizer.fit_on_texts(texts)
11
12print(tokenizer.word_index)
13print(tokenizer.texts_to_sequences(["apple banana pear grape"]))

A typical word_index might contain all four words even though num_words=3.

Why? Because fitting still records the whole vocabulary. But when texts_to_sequences runs, only words whose indices are inside the allowed range are emitted.

That is the part people often misunderstand.

Why the Word Index Still Looks Larger

Many developers inspect tokenizer.word_index and think num_words is not working because the dictionary contains more tokens than expected.

That behavior is normal.

The tokenizer stores all discovered words in word_index, but the cutoff is enforced when converting text. In practice, that means a rare word may appear in word_index and still be ignored during sequence generation.

So this is the right mental model:

• 'fit_on_texts() learns the full ranking'
• 'texts_to_sequences() applies the num_words cutoff'
• 'word_index is not automatically truncated'

About the Actual Index Limit

Keras token indices start at 1, while 0 is typically reserved for padding. As a result, the cutoff behaves like an index threshold, not like a raw dictionary length guarantee.

In practical terms, words with indices at or beyond the cutoff are filtered out during sequence conversion.

That is why num_words=10000 is usually described as “use only the most frequent words up to that limit,” not “store exactly 10000 entries in the word index.”

What Happens to Filtered-Out Words

There are two common outcomes for words outside the limit:

• they are dropped entirely if no out-of-vocabulary token is configured
• they map to the OOV token if you created the tokenizer with oov_token=...

Example:

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = ["red blue red green", "blue yellow green"]
4
5tokenizer = Tokenizer(num_words=3, oov_token="<OOV>")
6tokenizer.fit_on_texts(texts)
7
8sequence = tokenizer.texts_to_sequences(["red yellow blue black"])
9print(tokenizer.word_index)
10print(sequence)

With an OOV token, words outside the usable vocabulary do not disappear silently. They map to the OOV index instead, which is often better for production models.

Why You Would Limit Vocabulary

num_words is useful for several reasons:

• smaller embedding matrices
• less memory usage
• less noise from rare words
• faster training and inference
• better control of model size

A vocabulary with every rare typo and one-off token is often not helpful. Limiting the vocabulary can improve both performance and generalization.

Modern Context

The classic Tokenizer API is still widely seen in older tutorials and codebases, but many newer TensorFlow projects use TextVectorization instead. The conceptual question remains the same: how much vocabulary should the model keep?

So even if you migrate away from Tokenizer, understanding num_words helps you understand vocabulary cutoffs in general.

Common Pitfalls

The biggest pitfall is expecting word_index itself to shrink to exactly num_words entries. That is not how the API works.

Another issue is forgetting that the cutoff is frequency-based. Rare words are the ones that get pushed out, not words that merely appear later alphabetically.

Developers also sometimes set num_words but forget to configure an OOV token, which means out-of-range words disappear completely during sequence conversion.

Finally, if you pad sequences, remember that index 0 is usually reserved for padding and is not part of the learned vocabulary.

Summary

• 'num_words limits the usable vocabulary during text-to-sequence or matrix conversion.'
• The tokenizer still learns the full word ranking during fitting.
• 'word_index can contain more words than num_words, and that is normal.'
• Words outside the cutoff are dropped or mapped to an OOV token.
• The main reason to use num_words is to control vocabulary size, model size, and noise.