Introduction

A Keras tokenizer can only assign word indices for tokens it knows. When new words appear at inference time, they are either dropped or mapped to an out-of-vocabulary token, depending on how the tokenizer was configured. The right behavior is usually to keep a dedicated OOV bucket instead of silently losing information.

What Happens to Unseen Words by Default

When you fit a tokenizer, it builds a vocabulary from the training text only.

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = [
4    "cats chase mice",
5    "dogs chase balls"
6]
7
8tokenizer = Tokenizer()
9tokenizer.fit_on_texts(texts)
10print(tokenizer.word_index)

If you later encode text with a word the tokenizer never saw, that word is ignored.

sequence = tokenizer.texts_to_sequences(["cats chase birds"])
print(sequence)

birds is not in the vocabulary, so it disappears. That behavior is often undesirable because the model receives less information than expected.

Use oov_token to Keep Unknown Words Represented

The standard fix is to create the tokenizer with oov_token.

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = [
4    "cats chase mice",
5    "dogs chase balls"
6]
7
8tokenizer = Tokenizer(oov_token="[OOV]")
9tokenizer.fit_on_texts(texts)
10
11print(tokenizer.word_index)
12print(tokenizer.texts_to_sequences(["cats chase birds"]))

Now birds is mapped to the OOV token index instead of being discarded.

This is a much safer default for production NLP systems.

Why OOV Handling Matters

A model trained with one vocabulary will always encounter drift in real input:

• misspellings
• new product names
• slang
• rare domain terms
• user-generated text that was not present in training

If you let unknown words vanish, the model may behave unpredictably because the sequence becomes shorter or loses important cues.

An explicit OOV token tells the model, "this word was unknown," which is often better than pretending the word never existed.

num_words Also Creates OOV Behavior

Even known training words can become effectively unknown if you limit the vocabulary with num_words.

1from tensorflow.keras.preprocessing.text import Tokenizer
2
3texts = [
4    "red green blue",
5    "green blue yellow",
6    "blue yellow orange"
7]
8
9tokenizer = Tokenizer(num_words=3, oov_token="[OOV]")
10tokenizer.fit_on_texts(texts)
11
12print(tokenizer.word_index)
13print(tokenizer.texts_to_sequences(["red blue orange purple"]))

With num_words=3, only the most frequent tokens are preserved for sequence conversion. Less frequent words are treated as OOV even if they were present during fitting.

That is useful when you intentionally cap vocabulary size, but you should remember that OOV handling then covers both unseen words and low-frequency trimmed words.

Save and Reuse the Same Tokenizer

A common deployment mistake is refitting the tokenizer on new data and changing the word-to-index mapping. The model weights were trained against one exact mapping, so changing it breaks inference semantics.

Instead, fit once on training data and save the tokenizer.

1import json
2from tensorflow.keras.preprocessing.text import tokenizer_from_json
3
4config = tokenizer.to_json()
5with open("tokenizer.json", "w", encoding="utf-8") as f:
6    f.write(config)
7
8with open("tokenizer.json", "r", encoding="utf-8") as f:
9    loaded = tokenizer_from_json(f.read())

The serving system should load this saved tokenizer and use it unchanged.

Pad Sequences Consistently

Unknown words are only part of the preprocessing story. The final sequence length must still match what the model expects.

1from tensorflow.keras.preprocessing.sequence import pad_sequences
2
3seqs = tokenizer.texts_to_sequences(["cats chase birds quickly"])
4padded = pad_sequences(seqs, maxlen=6, padding="post", truncating="post")
5print(padded)

If training used padded sequences of a fixed length, inference must use the same padding and truncation rules.

OOV Tokens Are Helpful but Not Magic

An OOV token preserves structure, but it does not teach the model the meaning of a new word. All unknown words collapse to the same index. That is acceptable for many classical sequence models, but it limits nuance.

If your application depends heavily on unseen vocabulary, consider subword tokenization or transformer-based text preprocessing. Still, for many Keras pipelines built around word indices, oov_token is the correct baseline choice.

Tokenizer Is Fine, but Newer Pipelines Exist

The older Keras Tokenizer API still works, but newer projects often use TextVectorization or model families with built-in tokenizers. Those approaches can be easier to package and keep consistent across training and serving.

Even so, the core idea remains the same: unknown tokens need a deliberate policy.

Common Pitfalls

Fitting a tokenizer without oov_token and assuming unseen words will be handled automatically.

Refitting the tokenizer on production data, which changes the word indices the model expects.

Forgetting that num_words can turn low-frequency known words into OOV tokens.

Using different padding or truncation settings at inference than during training.

Assuming the OOV token gives semantic understanding of every unseen word. It only marks them as unknown.

Summary

• A Keras tokenizer only knows words seen during fitting unless you define OOV behavior.
• Without oov_token, unseen words are dropped from sequences.
• With oov_token, unknown and trimmed words map to a dedicated index.
• Save and reuse the original tokenizer instead of refitting it later.
• Keep padding and vocabulary settings identical between training and inference.