Introduction

Text rarely tells the whole story in a machine learning problem. A classifier can often improve when bag-of-words features are combined with metadata such as ratings, categories, message length, or customer source.

Why Mixed Feature Sets Help

Bag-of-words features capture which terms appear in a document, but they do not automatically include structured information that may also be predictive. Imagine a support-ticket classifier. The words in the ticket matter, but so do the product tier, ticket age, and whether the request came from email or chat.

In scikit-learn, the cleanest way to combine these signals is to keep your data in a pandas DataFrame and let a ColumnTransformer apply the right preprocessing to each column. Text can go through CountVectorizer or TfidfVectorizer, numeric columns can be scaled, and categorical columns can be one-hot encoded. The resulting matrices are then concatenated into a single feature matrix.

Building a Combined Pipeline

The following example trains a sentiment model from review text plus a numeric score and a categorical source column.

1import pandas as pd
2from sklearn.compose import ColumnTransformer
3from sklearn.feature_extraction.text import TfidfVectorizer
4from sklearn.linear_model import LogisticRegression
5from sklearn.metrics import classification_report
6from sklearn.model_selection import train_test_split
7from sklearn.pipeline import Pipeline
8from sklearn.preprocessing import OneHotEncoder, StandardScaler
9
10df = pd.DataFrame(
11    {
12        "review": [
13            "fast delivery and excellent packaging",
14            "late shipment and broken screen",
15            "works well for the price",
16            "customer support never replied",
17            "excellent battery life and great display",
18            "refund process was slow",
19        ],
20        "rating": [5, 1, 4, 1, 5, 2],
21        "source": ["web", "mobile", "web", "email", "mobile", "email"],
22        "label": [1, 0, 1, 0, 1, 0],
23    }
24)
25
26X_train, X_test, y_train, y_test = train_test_split(
27    df[["review", "rating", "source"]],
28    df["label"],
29    test_size=0.33,
30    random_state=42,
31)
32
33preprocessor = ColumnTransformer(
34    transformers=[
35        ("text", TfidfVectorizer(ngram_range=(1, 2), min_df=1), "review"),
36        ("num", StandardScaler(), ["rating"]),
37        ("cat", OneHotEncoder(handle_unknown="ignore"), ["source"]),
38    ]
39)
40
41model = Pipeline(
42    steps=[
43        ("features", preprocessor),
44        ("classifier", LogisticRegression(max_iter=1000)),
45    ]
46)
47
48model.fit(X_train, y_train)
49predictions = model.predict(X_test)
50
51print(classification_report(y_test, predictions))

This pattern matters because it keeps training and inference aligned. The same vocabulary, scaling parameters, and category mapping used during training are reused automatically when you call predict.

Working With Sparse and Dense Inputs

Text vectorizers normally produce a sparse matrix because most documents contain only a small fraction of the vocabulary. That is good for memory efficiency. Numeric and categorical features may be dense or sparse depending on the transformer.

When you use a ColumnTransformer, scikit-learn handles the concatenation for you. In many practical cases you do not need to manually call scipy.sparse.hstack. Manual stacking is still useful if you already computed the text matrix separately:

1import pandas as pd
2from scipy.sparse import hstack
3from sklearn.feature_extraction.text import CountVectorizer
4
5df = pd.DataFrame(
6    {
7        "text": ["red apple", "green apple", "blue car"],
8        "price": [3.2, 3.5, 19.0],
9    }
10)
11
12vectorizer = CountVectorizer()
13text_features = vectorizer.fit_transform(df["text"])
14
15numeric_features = df[["price"]].to_numpy()
16combined = hstack([text_features, numeric_features])
17
18print(combined.shape)

The pipeline approach is still preferable for production code because it reduces the chance of mismatched preprocessing steps.

When to Use Counts Versus TF-IDF

CountVectorizer gives raw term frequencies. It is simple and works well with models such as multinomial naive Bayes. TfidfVectorizer down-weights very common terms and is often a better default for linear models such as logistic regression or linear support vector machines.

The decision depends on both the dataset and the model. If your auxiliary features are strong, the exact text weighting scheme may matter less than having a reliable preprocessing pipeline. The important part is that all features are learned together from the same training split.

Common Pitfalls

• Fitting the vectorizer on the full dataset before the train-test split causes leakage and inflates evaluation metrics.
• Calling .toarray() on a large sparse text matrix can exhaust memory quickly.
• Forgetting to encode categorical columns will make most models fail because they cannot consume raw strings.
• Scaling numeric features separately from the saved pipeline can create train and inference mismatches.
• Misaligned pandas indices can silently pair the wrong rows if you preprocess text and tabular features in separate steps.

Summary

• Keep text and tabular data together in a pandas DataFrame so row alignment stays simple.
• Use ColumnTransformer to apply text, numeric, and categorical preprocessing in one place.
• Wrap preprocessing and the estimator in a Pipeline so training and inference share the same logic.
• Prefer sparse-aware workflows and avoid densifying text features unless the dataset is small.
• Choose CountVectorizer or TfidfVectorizer based on the model and dataset, then validate with cross-validation.