Practice Now
Keras
Tokenizer
Python
Deep Learning
Troubleshooting

Unable to import Tokenizer from Keras

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Deep LearningTensorFlow & KerasPythonDebugging & Errors

Introduction

The Tokenizer class was part of keras.preprocessing.text in older Keras/TensorFlow versions. In TensorFlow 2.16+ with Keras 3, Tokenizer was removed because it is a legacy preprocessing utility. The import from keras.preprocessing.text import Tokenizer fails with ImportError or ModuleNotFoundError. The fix depends on your TensorFlow version: use from tensorflow.keras.preprocessing.text import Tokenizer for TF 2.x (before Keras 3), or switch to tf.keras.layers.TextVectorization which is the modern replacement.

The Error

python
1# These imports may fail depending on your TF/Keras version:
2
3# Old standalone Keras import
4from keras.preprocessing.text import Tokenizer
5# ImportError: cannot import name 'Tokenizer' from 'keras.preprocessing.text'
6
7# Keras 3 import
8from keras.preprocessing.text import Tokenizer
9# ModuleNotFoundError: No module named 'keras.preprocessing.text'

Fix 1: Use tensorflow.keras (TF 2.0 - 2.15)

python
1# This works for TensorFlow 2.0 through 2.15
2from tensorflow.keras.preprocessing.text import Tokenizer
3
4tokenizer = Tokenizer(num_words=10000, oov_token="<OOV>")
5
6texts = [
7    "I love machine learning",
8    "Deep learning is amazing",
9    "Natural language processing with Python"
10]
11
12tokenizer.fit_on_texts(texts)
13
14# Convert text to sequences of integers
15sequences = tokenizer.texts_to_sequences(texts)
16print(sequences)
17# [[2, 3, 4, 1], [5, 1, 6, 7], [8, 9, 10, 11, 12]]
18
19# Word index mapping
20print(tokenizer.word_index)
21# {'learning': 1, 'i': 2, 'love': 3, 'machine': 4, 'deep': 5, ...}

tensorflow.keras is the integrated Keras that ships with TensorFlow. The Tokenizer class is available here through TF 2.15.

Fix 2: Install tf-keras for Backward Compatibility

bash
# For TF 2.16+ where Keras 3 is the default
pip install tf-keras
python
1# Import from tf-keras (the old Keras 2 API)
2import tf_keras as keras
3from tf_keras.preprocessing.text import Tokenizer
4
5tokenizer = Tokenizer(num_words=5000)
6tokenizer.fit_on_texts(["hello world", "goodbye world"])
7print(tokenizer.texts_to_sequences(["hello goodbye"]))

tf-keras is the Keras 2 API preserved as a separate package for backward compatibility with TF 2.16+.

python
1import tensorflow as tf
2
3# TextVectorization is the modern replacement for Tokenizer
4vectorize_layer = tf.keras.layers.TextVectorization(
5    max_tokens=10000,
6    output_mode="int",
7    output_sequence_length=20
8)
9
10# Adapt (fit) on training data
11train_texts = [
12    "I love machine learning",
13    "Deep learning is amazing",
14    "Natural language processing with Python"
15]
16
17vectorize_layer.adapt(train_texts)
18
19# Convert texts to integer sequences
20sequences = vectorize_layer(train_texts)
21print(sequences)
22# tf.Tensor([[4, 5, 6, 2, 0, ...], [7, 2, 8, 9, 0, ...], ...])
23
24# Get vocabulary
25print(vectorize_layer.get_vocabulary()[:10])
26# ['', '[UNK]', 'learning', ...]

TextVectorization is a Keras layer that can be included directly in the model, making the preprocessing part of the computation graph and exportable with the model.

TextVectorization in a Model

python
1import tensorflow as tf
2import numpy as np
3
4# Create the vectorization layer
5max_tokens = 10000
6sequence_length = 100
7
8vectorize_layer = tf.keras.layers.TextVectorization(
9    max_tokens=max_tokens,
10    output_mode="int",
11    output_sequence_length=sequence_length
12)
13
14# Adapt on training data
15train_texts = ["great movie", "terrible film", "loved it", "waste of time"]
16vectorize_layer.adapt(train_texts)
17
18# Build model with vectorization included
19model = tf.keras.Sequential([
20    tf.keras.Input(shape=(1,), dtype=tf.string),
21    vectorize_layer,
22    tf.keras.layers.Embedding(max_tokens, 64),
23    tf.keras.layers.GlobalAveragePooling1D(),
24    tf.keras.layers.Dense(64, activation="relu"),
25    tf.keras.layers.Dense(1, activation="sigmoid")
26])
27
28model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
29
30# Train — pass raw strings directly
31labels = np.array([1, 0, 1, 0])
32model.fit(np.array(train_texts), labels, epochs=5)
33
34# Predict — no preprocessing needed
35prediction = model.predict(np.array(["amazing film"]))
36print(prediction)

The model accepts raw text strings and handles tokenization internally. This makes deployment simpler because the preprocessing is part of the saved model.

Tokenizer vs TextVectorization

FeatureTokenizerTextVectorization
APIStandalone utilityKeras layer
Part of modelNo (preprocessing step)Yes (in the graph)
Exportable with modelNo (save separately)Yes (saved with model)
Keras 3 supportRemovedSupported
tf.data compatibleVia tf.py_functionNative
Fit methodfit_on_texts()adapt()

Padding Sequences (Companion Fix)

python
1# If you also used pad_sequences, it moved too:
2
3# Old import (may not work)
4# from keras.preprocessing.sequence import pad_sequences
5
6# TF 2.0 - 2.15
7from tensorflow.keras.preprocessing.sequence import pad_sequences
8
9# Modern alternative: use tf.keras.utils.pad_sequences
10from tensorflow.keras.utils import pad_sequences
11
12sequences = [[1, 2, 3], [4, 5], [6, 7, 8, 9]]
13padded = pad_sequences(sequences, maxlen=5, padding="post")
14print(padded)
15# [[1, 2, 3, 0, 0],
16#  [4, 5, 0, 0, 0],
17#  [6, 7, 8, 9, 0]]

Common Pitfalls

  • Importing from keras instead of tensorflow.keras: Standalone keras and tensorflow.keras are different packages. In TF 2.x, always use from tensorflow.keras.preprocessing.text import Tokenizer. The standalone keras package (Keras 3) removed Tokenizer.
  • Using Tokenizer with TF 2.16+ (Keras 3): Keras 3 removed keras.preprocessing.text.Tokenizer entirely. Install tf-keras for backward compatibility or migrate to TextVectorization.
  • Saving a Tokenizer separately from the model: Tokenizer is not part of the model graph, so saving the model does not save the tokenizer. You must serialize it separately with tokenizer.to_json() or pickle. TextVectorization is saved automatically with the model.
  • Not calling adapt() before using TextVectorization: Unlike Tokenizer.fit_on_texts(), TextVectorization.adapt() must be called on the training data before the layer can be used. Forgetting this raises an error about uninitialized vocabulary.
  • Mixing Keras 2 and Keras 3 imports: Having both keras (v3) and tf-keras (v2) installed can cause confusion. Import explicitly from tf_keras for Keras 2 APIs or from keras for Keras 3 APIs. Do not mix them in the same file.

Summary

  • Tokenizer was removed in Keras 3 (TF 2.16+) — use from tensorflow.keras.preprocessing.text import Tokenizer for TF 2.0-2.15
  • Install tf-keras for backward compatibility with TF 2.16+
  • Migrate to tf.keras.layers.TextVectorization as the modern replacement
  • TextVectorization is a Keras layer that can be included in the model and exported with it
  • pad_sequences also moved — use from tensorflow.keras.utils import pad_sequences
  • Always check your TensorFlow/Keras version to determine which import path to use
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.