Unable to import Tokenizer from Keras

Keras

Tokenizer

Python

Deep Learning

Troubleshooting

Unable to import Tokenizer from Keras

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

The Tokenizer class was part of keras.preprocessing.text in older Keras/TensorFlow versions. In TensorFlow 2.16+ with Keras 3, Tokenizer was removed because it is a legacy preprocessing utility. The import from keras.preprocessing.text import Tokenizer fails with ImportError or ModuleNotFoundError. The fix depends on your TensorFlow version: use from tensorflow.keras.preprocessing.text import Tokenizer for TF 2.x (before Keras 3), or switch to tf.keras.layers.TextVectorization which is the modern replacement.

The Error

python

1# These imports may fail depending on your TF/Keras version:
2
3# Old standalone Keras import
4from keras.preprocessing.text import Tokenizer
5# ImportError: cannot import name 'Tokenizer' from 'keras.preprocessing.text'
6
7# Keras 3 import
8from keras.preprocessing.text import Tokenizer
9# ModuleNotFoundError: No module named 'keras.preprocessing.text'

Fix 1: Use tensorflow.keras (TF 2.0 - 2.15)

python

1# This works for TensorFlow 2.0 through 2.15
2from tensorflow.keras.preprocessing.text import Tokenizer
3
4tokenizer = Tokenizer(num_words=10000, oov_token="<OOV>")
5
6texts = [
7    "I love machine learning",
8    "Deep learning is amazing",
9    "Natural language processing with Python"
10]
11
12tokenizer.fit_on_texts(texts)
13
14# Convert text to sequences of integers
15sequences = tokenizer.texts_to_sequences(texts)
16print(sequences)
17# [[2, 3, 4, 1], [5, 1, 6, 7], [8, 9, 10, 11, 12]]
18
19# Word index mapping
20print(tokenizer.word_index)
21# {'learning': 1, 'i': 2, 'love': 3, 'machine': 4, 'deep': 5, ...}

tensorflow.keras is the integrated Keras that ships with TensorFlow. The Tokenizer class is available here through TF 2.15.

Fix 2: Install tf-keras for Backward Compatibility

bash

# For TF 2.16+ where Keras 3 is the default
pip install tf-keras

python

1# Import from tf-keras (the old Keras 2 API)
2import tf_keras as keras
3from tf_keras.preprocessing.text import Tokenizer
4
5tokenizer = Tokenizer(num_words=5000)
6tokenizer.fit_on_texts(["hello world", "goodbye world"])
7print(tokenizer.texts_to_sequences(["hello goodbye"]))

tf-keras is the Keras 2 API preserved as a separate package for backward compatibility with TF 2.16+.

Fix 3: Use TextVectorization Layer (Recommended)

python

1import tensorflow as tf
2
3# TextVectorization is the modern replacement for Tokenizer
4vectorize_layer = tf.keras.layers.TextVectorization(
5    max_tokens=10000,
6    output_mode="int",
7    output_sequence_length=20
8)
9
10# Adapt (fit) on training data
11train_texts = [
12    "I love machine learning",
13    "Deep learning is amazing",
14    "Natural language processing with Python"
15]
16
17vectorize_layer.adapt(train_texts)
18
19# Convert texts to integer sequences
20sequences = vectorize_layer(train_texts)
21print(sequences)
22# tf.Tensor([[4, 5, 6, 2, 0, ...], [7, 2, 8, 9, 0, ...], ...])
23
24# Get vocabulary
25print(vectorize_layer.get_vocabulary()[:10])
26# ['', '[UNK]', 'learning', ...]

TextVectorization is a Keras layer that can be included directly in the model, making the preprocessing part of the computation graph and exportable with the model.

TextVectorization in a Model

python

1import tensorflow as tf
2import numpy as np
3
4# Create the vectorization layer
5max_tokens = 10000
6sequence_length = 100
7
8vectorize_layer = tf.keras.layers.TextVectorization(
9    max_tokens=max_tokens,
10    output_mode="int",
11    output_sequence_length=sequence_length
12)
13
14# Adapt on training data
15train_texts = ["great movie", "terrible film", "loved it", "waste of time"]
16vectorize_layer.adapt(train_texts)
17
18# Build model with vectorization included
19model = tf.keras.Sequential([
20    tf.keras.Input(shape=(1,), dtype=tf.string),
21    vectorize_layer,
22    tf.keras.layers.Embedding(max_tokens, 64),
23    tf.keras.layers.GlobalAveragePooling1D(),
24    tf.keras.layers.Dense(64, activation="relu"),
25    tf.keras.layers.Dense(1, activation="sigmoid")
26])
27
28model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
29
30# Train — pass raw strings directly
31labels = np.array([1, 0, 1, 0])
32model.fit(np.array(train_texts), labels, epochs=5)
33
34# Predict — no preprocessing needed
35prediction = model.predict(np.array(["amazing film"]))
36print(prediction)

The model accepts raw text strings and handles tokenization internally. This makes deployment simpler because the preprocessing is part of the saved model.

Tokenizer vs TextVectorization

Feature	Tokenizer	TextVectorization
API	Standalone utility	Keras layer
Part of model	No (preprocessing step)	Yes (in the graph)
Exportable with model	No (save separately)	Yes (saved with model)
Keras 3 support	Removed	Supported
`tf.data` compatible	Via `tf.py_function`	Native
Fit method	`fit_on_texts()`	`adapt()`

Padding Sequences (Companion Fix)

python

1# If you also used pad_sequences, it moved too:
2
3# Old import (may not work)
4# from keras.preprocessing.sequence import pad_sequences
5
6# TF 2.0 - 2.15
7from tensorflow.keras.preprocessing.sequence import pad_sequences
8
9# Modern alternative: use tf.keras.utils.pad_sequences
10from tensorflow.keras.utils import pad_sequences
11
12sequences = [[1, 2, 3], [4, 5], [6, 7, 8, 9]]
13padded = pad_sequences(sequences, maxlen=5, padding="post")
14print(padded)
15# [[1, 2, 3, 0, 0],
16#  [4, 5, 0, 0, 0],
17#  [6, 7, 8, 9, 0]]

Common Pitfalls

Importing from keras instead of tensorflow.keras: Standalone keras and tensorflow.keras are different packages. In TF 2.x, always use from tensorflow.keras.preprocessing.text import Tokenizer. The standalone keras package (Keras 3) removed Tokenizer.
Using Tokenizer with TF 2.16+ (Keras 3): Keras 3 removed keras.preprocessing.text.Tokenizer entirely. Install tf-keras for backward compatibility or migrate to TextVectorization.
Saving a Tokenizer separately from the model: Tokenizer is not part of the model graph, so saving the model does not save the tokenizer. You must serialize it separately with tokenizer.to_json() or pickle. TextVectorization is saved automatically with the model.
Not calling adapt() before using TextVectorization: Unlike Tokenizer.fit_on_texts(), TextVectorization.adapt() must be called on the training data before the layer can be used. Forgetting this raises an error about uninitialized vocabulary.
Mixing Keras 2 and Keras 3 imports: Having both keras (v3) and tf-keras (v2) installed can cause confusion. Import explicitly from tf_keras for Keras 2 APIs or from keras for Keras 3 APIs. Do not mix them in the same file.

Summary

Tokenizer was removed in Keras 3 (TF 2.16+) — use from tensorflow.keras.preprocessing.text import Tokenizer for TF 2.0-2.15
Install tf-keras for backward compatibility with TF 2.16+
Migrate to tf.keras.layers.TextVectorization as the modern replacement
TextVectorization is a Keras layer that can be included in the model and exported with it
pad_sequences also moved — use from tensorflow.keras.utils import pad_sequences
Always check your TensorFlow/Keras version to determine which import path to use