Introduction

Naive Bayes is a probabilistic classifier based on applying Bayes’ theorem with strong independence assumptions between the features. It is particularly popular for text classification due to its simplicity and effectiveness. One of the core components of the Naive Bayes classifier is the calculation of probabilities, for which the log likelihood is often used to improve computational efficiency and numerical stability. This article explores the concept of log likelihood in the context of implementing Naive Bayes for text classification.

Understanding Naive Bayes

Naive Bayes relies on the assumption that the presence (or absence) of a particular feature in a class is independent of the presence (or absence) of any other feature. For text classification, this means the model assumes that the presence of any word in a document is independent of the presence of any other word.

The key principle behind Naive Bayes is Bayes' theorem, which can be expressed as:

$$P(C \mid X) = \frac{P(X \mid C) P(C)}{P(X)}$$

Where: • $P(C \mid X)$: Posterior probability of class $C$ given predictor $X$. • $P(X \mid C)$: Likelihood of predictor $X$ given class $C$. • $P(C)$: Prior probability of class $C$. • $P(X)$: Prior probability of predictor $X$.

Log Likelihood in Naive Bayes

Why Use Log Likelihood?

When implementing Naive Bayes, especially in text classification, the likelihood component $P(X \mid C)$ can become very small when dealing with long documents or large vocabularies. Since multiplying small probabilities leads to numerical underflow, we use the log likelihood to convert these products into sums, which are easier to compute and more stable.

Log Likelihood Formula

The log likelihood in Naive Bayes is calculated as the sum of the log probabilities of the individual features. For a document $d$ with words $w_1, w_2, \ldots, w_n$, and class $C$, the log likelihood can be computed as:

$$\log P(d \mid C) = \log \prod\_{i=1}^{n} P(w\_i \mid C) = \sum\_{i=1}^{n} \log P(w\_i \mid C)$$

This conversion not only helps in preventing underflow but also allows vectorizing the computation, which is more efficient.

Using Log Likelihood in Classification

To classify a new text sample, you calculate the posterior probabilities for each class (in log form) and select the class with the highest probability:

$$\log P(C \mid d) = \log P(C) + \log P(d \mid C)$$

Where $\log P(d \mid C)$ is the log likelihood we computed earlier.

Example of Implementing Naive Bayes with Log Likelihood

Consider a simplified vocabulary and two classes, Positive and Negative . Suppose we have the following computed probabilities (for demonstration):

Let's classify a new document: "Good Neutral".

1. Calculate Log Likelihood for Each Class:
   • For Positive: $$\log P(\text{"Good Neutral"} \mid \text{Positive}) = \log(0.8) + \log(0.1)$$ $$= \log P(\text{"Good"} \mid \text{Positive}) + \log P(\text{"Neutral"} \mid \text{Positive})$$ $$= -0.22 + (-2.30) = -2.52$$
   • For Negative: $$\log P(\text{"Good Neutral"} \mid \text{Negative}) = \log(0.2) + \log(0.2)$$ $$= \log P(\text{"Good"} \mid \text{Negative}) + \log P(\text{"Neutral"} \mid \text{Negative})$$ $$= -1.61 + (-1.61) = -3.22$$
2. Include Log Priors and Choose Class:
   Assume equal priors for simplicity: $\log P(\text{Positive}) = \log P(\text{Negative}) = \log(0.5) = -0.69$.
   • $\log P(\text{Positive} \mid \text{"Good Neutral"}) = -0.69 - 2.52 = -3.21$ • $\log P(\text{Negative} \mid \text{"Good Neutral"}) = -0.69 - 3.22 = -3.91$

The document "Good Neutral" is classified as Positive because $-3.21 > -3.91$.

Summary Table

Additional Considerations

Laplace Smoothing

In practice, the probability $P(w_i \mid C)$ can be zero if a word has not appeared in a training sample for a class. To handle this, Laplace smoothing is often applied to avoid zero probabilities by adding a small constant to each count before calculating probabilities.

Advantages of Using Log Likelihood

• Numerical Stability: Converting probabilities to logs helps manage calculations with very small numbers. • Efficiency: Logarithms allow probabilities to be added rather than multiplied, which can be computationally cheaper. • Simplicity: Maintains the simplicity of Naive Bayes while enabling more robust performance.

Overall, log likelihood is an integral part of implementing Naive Bayes for text classification, ensuring robustness and efficiency while maintaining simplicity in the classification process.