bigram
unigram
text features
natural language processing
machine learning

what is the difference between bigram and unigram text features extraction

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Introduction

In the realm of natural language processing (NLP) and text analysis, feature extraction is a critical step for converting textual content into numerical representations that machine learning algorithms can effectively process. Among the foundational techniques for text feature extraction are unigram and bigram models. Understanding the differences between these methods is crucial for selecting the right approach for a given application.

Understanding Unigrams and Bigrams

Unigrams

A unigram is a single word or token in a text. When text features are extracted using a unigram model, each word in the corpus is treated as an independent feature. This means the model considers the frequency of each word without regard to any contextual information from surrounding words.

Example

Consider the sentence: "The cat sat on the mat."

• Unigrams: `["The", "cat", "sat", "on", "the", "mat"]`

In this case, each word becomes a feature, and the text representation focuses on individual word occurrences.

Bigrams

A bigram consists of two consecutive words (tokens) in a text. A bigram model captures some level of context by considering word pairs together. This can be particularly useful for detecting common phrases or sequences words often appear in.

Example

Using the same sentence: "The cat sat on the mat."

• Bigrams: `[("The", "cat"), ("cat", "sat"), ("sat", "on"), ("on", "the"), ("the", "mat")]`

Here, each pair of words is treated as a feature, capturing relationships between adjacent words.

Technical Explanation

Unigram Model

In the unigram model, each sentence is transformed into a sparse vector where the dimensionality equals the vocabulary size. The vector comprises counts (or frequencies) of each word in the sentence with respect to the whole dataset.

Mathematical Representation

Suppose the vocabulary V=w1,w2,,wnV = {w_1, w_2, \ldots, w_n}. For any given document DD, a unigram feature vector Funi(D)F_{\text{uni}}(D) can be represented as:

F_uni(D)=[f(w_1,D),f(w_2,D),,f(w_n,D)]F\_{\text{uni}}(D) = [f(w\_1, D), f(w\_2, D), \ldots, f(w\_n, D)]

where f(wi,D)f(w_i, D) signifies the frequency of word wiw_i in document DD.

Bigram Model

In the bigram model, the focus shifts to sequences of two consecutive words. Similar to unigrams, a bigram vector represents the frequency of each bigram in the document.

Mathematical Representation

For the vocabulary VV, a bigram vocabulary is constructed as V=(wi,wj)wi,wjVV' = {(w_i, w_j) \mid w_i, w_j \in V}. For a document DD, the bigram feature vector Fbi(D)F_{\text{bi}}(D) is:

F_bi(D)=[f((w_1,w_2),D),f((w_2,w_3),D),]F\_{\text{bi}}(D) = [f((w\_1, w\_2), D), f((w\_2, w\_3), D), \ldots]

where f((wi,wj),D)f((w_i, w_j), D) denotes the frequency of bigram (wi,wj)(w_i, w_j) in document DD.

Comparison Table

Feature Extraction MethodDefinitionContextual InformationExampleComputational Complexity
UnigramSingle word as featureMinimal"The", "cat"Lower: Linear in terms of number of words
BigramPair of words as featureModerate"The cat", "sat on"Higher: Linear in terms of number of word pairs

Applications and Considerations

Applications

Unigrams: Suitable for applications where individual word importance prevails, such as simple document classification or sentiment analysis based on single-word sentiment lexicons. • Bigrams: Ideal for capturing more contextual nuances, as in phrase detection, more nuanced sentiment analysis, and language modeling where certain word combinations carry specific meanings.

Considerations

Sparsity: Bigram models result in a much larger feature space, which may lead to increased sparsity and require more data to train effectively. • Interpretability: Unigram models are often more interpretable as they directly map to individual words. • Performance: Depending on the dataset and specific task, the choice between unigram and bigram can significantly affect the performance. For instance, in tasks benefiting from phrase detection, bigrams can outperform unigrams.

Conclusion

Understanding the differences between unigram and bigram feature extraction methods allows for more informed decisions in text processing tasks. While unigrams focus on individual word occurrences, bigrams offer insight into word sequences and context. Each approach has its strengths and is best suited for different types of text analysis problems, making feature selection a paramount step in any NLP workflow.


Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

All Rights Reserved.