Introduction

The most important design point here is that TensorFlow Transform usually should not create dense word vectors directly. Its best role is to build stable token preprocessing and vocabulary mapping, while the actual conversion from token IDs to vectors is typically handled later by an embedding layer inside the model.

What TensorFlow Transform Is Good At

TensorFlow Transform, often called TFT, is built for full-pass preprocessing over training data. That makes it a good fit for:

• token cleanup and normalization
• vocabulary creation
• mapping tokens to integer IDs
• keeping training and serving preprocessing consistent

It is not primarily an embedding library. If you try to push dense vector logic into TFT itself, the pipeline often becomes harder to maintain than it needs to be.

The Normal Pattern: Tokens to IDs in TFT

The usual workflow is:

1. tokenize text
2. use TFT to compute and apply a vocabulary
3. feed integer IDs into a TensorFlow or Keras embedding layer

Here is a simplified preprocessing function:

1import tensorflow_transform as tft
2
3def preprocessing_fn(inputs):
4    tokens = inputs["tokens"]
5    token_ids = tft.compute_and_apply_vocabulary(
6        tokens,
7        top_k=20000,
8        num_oov_buckets=1
9    )
10    return {"token_ids": token_ids}

This gives you stable integer representations that can be used at training time and later reused at serving through the exported transform graph.

Turn Token IDs into Vectors in the Model

Once TFT has produced token IDs, the model learns or applies dense embeddings:

1import tensorflow as tf
2
3token_ids = tf.keras.Input(shape=(None,), dtype=tf.int64, name="token_ids")
4embedded = tf.keras.layers.Embedding(
5    input_dim=20001,
6    output_dim=128
7)(token_ids)
8
9pooled = tf.reduce_mean(embedded, axis=1)
10outputs = tf.keras.layers.Dense(1, activation="sigmoid")(pooled)
11
12model = tf.keras.Model(inputs=token_ids, outputs=outputs)

This split is effective because TFT handles deterministic preprocessing, while the model handles trainable representation learning.

Why This Split Works Better

Dense embeddings are model parameters. They belong in the training graph because they are learned, updated, checkpointed, and versioned with the model. Vocabulary mapping is preprocessing metadata. It belongs in TFT because it is derived from the training corpus and must stay identical between training and serving.

That separation gives you:

• reproducible token-to-index mapping
• trainable embeddings without awkward preprocessing hacks
• cleaner serving pipelines

In other words, TFT prepares the lookup key, and the model learns the vector space.

Using Pretrained Word Vectors

If you already have pretrained embeddings such as GloVe or FastText, the same structure still applies. Use TFT to map tokens to IDs, then initialize an embedding layer with a pretrained matrix.

1embedding_layer = tf.keras.layers.Embedding(
2    input_dim=vocab_size,
3    output_dim=300,
4    weights=[pretrained_matrix],
5    trainable=False
6)

You can later decide whether to keep those weights frozen or fine-tune them. The important point is that the vector lookup still belongs in the model, not in TFT's corpus-wide analysis step.

Sequence Features and Ragged Inputs

NLP pipelines often work with token sequences of different lengths. That means the representation coming out of TFT may be sparse, ragged, or padded depending on the pipeline design. Keep that in mind when building the model input signature.

The easiest approach is to make the preprocessing output explicit and stable, such as a padded sequence of token IDs or a ragged tensor that downstream layers know how to consume.

Common Pitfalls

• Trying to build dense word vectors inside TFT instead of inside the model.
• Mixing vocabulary generation and embedding weights as if they were the same artifact.
• Forgetting out-of-vocabulary handling with num_oov_buckets or a reserved token.
• Changing the vocabulary at serving time and breaking consistency with training.
• Treating pretrained embeddings as preprocessing metadata instead of model parameters.

Summary

• TFT is best used for token normalization and token-to-ID mapping.
• Dense word vectors usually belong in an embedding layer in the model.
• 'tft.compute_and_apply_vocabulary is the key bridge between tokens and trainable embeddings.'
• Pretrained vectors still fit the same pattern: TFT for IDs, model for vector lookup.
• Keeping preprocessing and learned representation separate makes training and serving cleaner.