Introduction

The clean path from a Pandas dataframe to TensorFlow NLP training is to use Pandas only for tabular cleanup, then switch into tf.data as early as possible. Trying to treat a dataframe itself like a training dataset usually leads to type issues, missing-value bugs, and slow input pipelines. A better pipeline is: clean dataframe columns, convert to tensors or arrays, build a dataset, vectorize text, then batch and prefetch.

Start With a Clean DataFrame

Assume a dataframe with one text column and one label column:

1import pandas as pd
2
3df = pd.DataFrame(
4    {
5        "text": [
6            "I loved this movie",
7            "Terrible pacing and weak acting",
8            "Surprisingly funny and warm",
9            "I would not watch it again",
10        ],
11        "label": [1, 0, 1, 0],
12    }
13)

Before handing anything to TensorFlow, normalize the columns you need:

df = df.dropna(subset=["text", "label"]).copy()
df["text"] = df["text"].astype(str)
df["label"] = df["label"].astype("int32")

That avoids common issues with NaN, mixed object types, and labels silently ending up as strings.

Build a tf.data.Dataset

The modern API is tf.data.Dataset. The older phrase BatchDataset usually just means a dataset after .batch() has been applied.

1import tensorflow as tf
2
3text_ds = tf.data.Dataset.from_tensor_slices(
4    (
5        df["text"].to_numpy(),
6        df["label"].to_numpy(),
7    )
8)
9
10text_ds = text_ds.shuffle(len(df)).batch(2).prefetch(tf.data.AUTOTUNE)

At this point, each batch contains raw strings and labels. That is already a valid TensorFlow input pipeline, but most NLP models still need tokenization or vectorization before training.

Add Text Vectorization

TextVectorization is a good default for many text-classification pipelines.

1vectorizer = tf.keras.layers.TextVectorization(
2    max_tokens=10000,
3    output_mode="int",
4    output_sequence_length=20,
5)
6
7vectorizer.adapt(df["text"].to_numpy())

Now map the dataset so text batches become integer token sequences:

1def vectorize_batch(text, label):
2    return vectorizer(text), label
3
4train_ds = text_ds.map(vectorize_batch).prefetch(tf.data.AUTOTUNE)
5
6for features, labels in train_ds.take(1):
7    print(features.shape)  # (batch_size, sequence_length)
8    print(labels.shape)    # (batch_size,)

This is often enough for sentiment analysis, spam detection, and small topic classifiers.

Full Example With a Keras Model

Here is a minimal end-to-end example:

1import pandas as pd
2import tensorflow as tf
3
4df = pd.DataFrame(
5    {
6        "text": [
7            "great acting and direction",
8            "boring script and slow ending",
9            "smart and entertaining",
10            "bad dialogue and weak plot",
11        ],
12        "label": [1, 0, 1, 0],
13    }
14)
15
16ds = tf.data.Dataset.from_tensor_slices(
17    (df["text"].astype(str).to_numpy(), df["label"].astype("int32").to_numpy())
18)
19ds = ds.shuffle(len(df)).batch(2)
20
21vectorizer = tf.keras.layers.TextVectorization(
22    max_tokens=5000,
23    output_mode="int",
24    output_sequence_length=16,
25)
26vectorizer.adapt(df["text"].to_numpy())
27
28ds = ds.map(lambda text, label: (vectorizer(text), label))
29ds = ds.prefetch(tf.data.AUTOTUNE)
30
31model = tf.keras.Sequential(
32    [
33        tf.keras.layers.Embedding(input_dim=5000, output_dim=16),
34        tf.keras.layers.GlobalAveragePooling1D(),
35        tf.keras.layers.Dense(16, activation="relu"),
36        tf.keras.layers.Dense(1, activation="sigmoid"),
37    ]
38)
39
40model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
41model.fit(ds, epochs=3)

The important part is that the dataframe is just the source of truth for the columns. Training happens through the tf.data pipeline, not through Pandas directly.

Multiple Inputs or Structured Features

If the model consumes text plus metadata, pass a feature dictionary instead of a single array:

1features = {
2    "title": df["text"].to_numpy(),
3    "length": df["text"].str.len().to_numpy(),
4}
5
6labels = df["label"].to_numpy()
7
8ds = tf.data.Dataset.from_tensor_slices((features, labels)).batch(32)

That is useful when text is combined with metadata such as message length, language, or author features.

Common Pitfalls

• Feeding dataframe rows directly into the model. Fix: convert required columns into arrays or datasets before training.
• Forgetting to clean NaN values. Fix: drop or fill missing text and label values before from_tensor_slices.
• Adapting TextVectorization on transformed tensors instead of raw text. Fix: adapt on the original text column.
• Losing label dtype or shape. Fix: cast labels deliberately to match the loss function you plan to use.
• Doing heavy Python preprocessing after batching. Fix: keep expensive transformations in the tf.data pipeline where possible.

Summary

• Use Pandas for column cleanup, then switch to tf.data.Dataset for training.
• Build the dataset with from_tensor_slices, then shuffle, batch, and prefetch.
• For NLP, vectorize text with TextVectorization or your tokenizer of choice.
• Adapt the vectorizer on raw text, not on already-batched token ids.
• Keep features and labels in consistent dtypes before handing them to TensorFlow.