Introduction

Combining TF-IDF with non-text features is a common pattern in text classification, ranking, and fraud-detection models. The correct approach is to keep the text pipeline sparse, preprocess the other features appropriately, and combine everything in one training pipeline so the model sees a single feature matrix.

The Two Main Ways to Combine Features

In scikit-learn, the cleanest method is usually ColumnTransformer, especially when the input is a DataFrame containing text, numeric, and categorical columns.

The older manual approach is to compute TF-IDF separately and then concatenate matrices with scipy.sparse.hstack. That still works, but it is easier to make training and inference inconsistent if preprocessing is spread across multiple steps.

A Practical ColumnTransformer Example

The example below combines one text column with two numeric features. TF-IDF handles the text, while numeric features are scaled separately.

1import pandas as pd
2from sklearn.compose import ColumnTransformer
3from sklearn.feature_extraction.text import TfidfVectorizer
4from sklearn.linear_model import LogisticRegression
5from sklearn.pipeline import Pipeline
6from sklearn.preprocessing import StandardScaler
7
8X = pd.DataFrame(
9    {
10        "text": [
11            "buy cheap pills now",
12            "meeting moved to tomorrow",
13            "limited time offer",
14            "project status update",
15        ],
16        "message_length": [20, 26, 18, 21],
17        "num_links": [3, 0, 2, 0],
18    }
19)
20y = [1, 0, 1, 0]
21
22preprocessor = ColumnTransformer(
23    transformers=[
24        ("text", TfidfVectorizer(), "text"),
25        ("num", StandardScaler(), ["message_length", "num_links"]),
26    ]
27)
28
29model = Pipeline(
30    steps=[
31        ("preprocessor", preprocessor),
32        ("classifier", LogisticRegression(max_iter=1000)),
33    ]
34)
35
36model.fit(X, y)
37print(model.predict(X))

This is the right pattern for production because the same fitted pipeline transforms future inputs in exactly the same way.

Why Sparse Data Matters

TF-IDF matrices are usually sparse, meaning most entries are zero. That is good. It keeps memory use manageable for large vocabularies.

When you add dense numeric features, scikit-learn can still combine them with the sparse text matrix. The important detail is to avoid operations that accidentally densify the full text matrix, because that can explode memory usage.

For that reason, tree-based models or linear models that accept sparse input are often good fits.

Manual Concatenation With hstack

If your data is already split across custom preprocessing steps, you can combine features manually.

1import numpy as np
2from scipy.sparse import hstack
3from sklearn.feature_extraction.text import TfidfVectorizer
4from sklearn.linear_model import LogisticRegression
5
6texts = [
7    "buy cheap pills now",
8    "meeting moved to tomorrow",
9    "limited time offer",
10    "project status update",
11]
12extra = np.array([
13    [20, 3],
14    [26, 0],
15    [18, 2],
16    [21, 0],
17], dtype="float32")
18y = [1, 0, 1, 0]
19
20vectorizer = TfidfVectorizer()
21X_text = vectorizer.fit_transform(texts)
22X_all = hstack([X_text, extra])
23
24clf = LogisticRegression(max_iter=1000)
25clf.fit(X_all, y)
26print(clf.predict(X_all))

That code is valid, but you now have to remember to apply the exact same vectorizer and feature ordering at inference time.

Which Extra Features Work Well

Useful non-text features often include:

• message length
• link count or attachment count
• language or source metadata
• author reputation scores
• categorical flags encoded numerically

The right additions depend on the task. TF-IDF captures token importance. The extra features capture signals that plain word counts miss.

Scaling and Model Choice

Numeric features often benefit from scaling, especially for linear models, logistic regression, and neural networks. TF-IDF values are already normalized by their own pipeline, so scale the non-text features separately rather than trying to normalize the combined matrix by hand.

If you use categorical metadata, encode it with OneHotEncoder in the same ColumnTransformer.

Common Pitfalls

The biggest mistake is fitting TF-IDF on the full dataset before the train-test split. That leaks information from the test set into the vocabulary and IDF statistics.

Another mistake is converting a huge sparse matrix to dense form. That can turn a tractable problem into an out-of-memory error.

A third issue is losing feature alignment. If you manually concatenate matrices, the training and inference column order must match exactly.

Finally, do not assume extra features always help. Some metadata is noisy and can hurt generalization unless validated properly.

Summary

• Use ColumnTransformer when combining TF-IDF with numeric or categorical features.
• Keep text features sparse and preprocess other columns separately.
• 'hstack works, but it is easier to make inference inconsistent.'
• Scale numeric features when the downstream model benefits from it.
• Avoid data leakage by fitting TF-IDF only on training data.
• Validate whether the extra features add signal instead of just complexity.