Introduction

Bigrams can improve text models because they capture short phrase context that unigrams miss, such as not good or credit card. The downside is that bigrams increase feature count quickly, so feature selection becomes essential if you want a model that is both accurate and efficient.

Why bigrams help

A unigram model treats words independently. That is often enough for topic classification, but it can lose important local structure. Bigrams help recover some of that structure.

Examples:

• 'new york means more than new plus york'
• 'not bad often reverses the sentiment of bad'
• 'machine learning is a phrase, not just two unrelated tokens'

That extra context is why bigrams are commonly used in sentiment analysis, topic labeling, and document classification.

Generate bigram features first

In scikit-learn, the simplest way to include bigrams is through CountVectorizer or TfidfVectorizer with ngram_range=(1, 2) or (2, 2).

1from sklearn.feature_extraction.text import TfidfVectorizer
2
3docs = [
4    "this movie is not good",
5    "this movie is very good",
6    "credit card fraud detection",
7    "machine learning pipeline",
8]
9
10vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1)
11X = vectorizer.fit_transform(docs)
12
13print(X.shape)
14print(vectorizer.get_feature_names_out()[:15])

This creates unigram and bigram features. If you want only bigrams, set ngram_range=(2, 2).

Then reduce the feature space

Once bigrams are generated, select features with a method that matches the task.

Common options:

• frequency thresholds such as min_df
• chi-squared selection for classification
• mutual information
• model-based sparsity such as L1 regularization

The fastest first filter is usually not a formal selector at all. It is just vocabulary pruning through min_df and max_df.

1from sklearn.feature_extraction.text import CountVectorizer
2
3vectorizer = CountVectorizer(
4    ngram_range=(1, 2),
5    min_df=2,
6    max_df=0.9,
7)

This removes rare phrases that are probably noise and extremely common phrases that add little discrimination.

Use chi-squared to keep the most informative bigrams

For text classification with nonnegative features, chi-squared is a practical selector.

1from sklearn.feature_extraction.text import TfidfVectorizer
2from sklearn.feature_selection import SelectKBest, chi2
3from sklearn.pipeline import Pipeline
4from sklearn.linear_model import LogisticRegression
5
6docs = [
7    "not good at all",
8    "really good experience",
9    "bad service and rude staff",
10    "very good and helpful",
11    "not helpful and not friendly",
12    "good service overall",
13]
14y = [0, 1, 0, 1, 0, 1]
15
16model = Pipeline([
17    ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
18    ("select", SelectKBest(score_func=chi2, k=8)),
19    ("clf", LogisticRegression(max_iter=1000)),
20])
21
22model.fit(docs, y)
23print(model.score(docs, y))

This keeps only the highest-scoring features after vectorization, which is often enough to tame a noisy bigram space.

Inspect which phrases survive

Feature selection is easier to trust when you inspect the retained terms.

1tfidf = TfidfVectorizer(ngram_range=(1, 2))
2X = tfidf.fit_transform(docs)
3
4selector = SelectKBest(chi2, k=8)
5X_selected = selector.fit_transform(X, y)
6
7all_features = tfidf.get_feature_names_out()
8selected_features = all_features[selector.get_support()]
9
10print(selected_features)
11print(X_selected.shape)

This is especially useful when reviewing whether the model is picking meaningful phrases or just artifacts from the dataset.

Bigrams are useful, but not always necessary

Do not assume bigrams automatically improve every task. They help most when phrase meaning matters. For some problems, a unigram model with solid preprocessing performs just as well and is easier to deploy.

A practical workflow is:

1. build a unigram baseline
2. add bigrams
3. prune features
4. compare validation metrics and model size

If the bigram model is only slightly better but much larger, the simpler baseline may be the better production choice.

Common Pitfalls

The most common mistake is enabling bigrams without pruning the vocabulary, which creates a huge sparse matrix and slows everything down. Another frequent issue is selecting features on the full dataset before the train and validation split, which leaks information and inflates metrics. Teams also sometimes assume every high-scoring bigram is semantically meaningful, when some are just dataset artifacts. Using only rare bigrams is another problem because they often look informative in training but fail to generalize. Finally, people often skip a unigram baseline, so they cannot tell whether bigrams actually helped.

Summary

• Bigrams capture short phrase context that unigrams often miss.
• Generate them with CountVectorizer or TfidfVectorizer using n-gram settings.
• Control feature explosion early with min_df, max_df, and vocabulary pruning.
• Use selectors such as chi-squared to keep the most informative terms.
• Inspect the retained phrases instead of treating feature selection as a black box.
• Always compare against a unigram baseline before deciding the extra complexity is worth it.