Introduction

Understanding document similarity is crucial in various applications, including search engines, recommendation systems, and content clustering. One of the popular techniques to determine the similarity between text documents is by leveraging TF-IDF and cosine similarity. In this article, we'll dive into these concepts and demonstrate how to implement them in Python.

What is TF-IDF?

TF-IDF stands for Term Frequency-Inverse Document Frequency. It is a numerical statistic that reflects the importance of a word in a document relative to a collection of documents, also known as a corpus.

Term Frequency (TF)

Term Frequency measures how frequently a word appears in a document. It's calculated using the formula:

$$\text{TF}(t, d) = \frac{f_{t,d}}{N_d}$$ Where:

• $f_{t,d}$ is the number of times term $t$ appears in document $d$.
• $N_d$ is the total number of terms in document $d$.

Inverse Document Frequency (IDF)

Inverse Document Frequency measures how important a term is universally across documents. Words like "is", "and", "the" may frequently appear, but it might not hold much importance. IDF is calculated as:

$$\text{IDF}(t, D) = \log\left(\frac{|D|}{1 + |{d \in D : t \in d}|}\right)$$ Where:

• $|D|$ is the total number of documents.
• $|{d \in D : t \in d}|$ is the number of documents containing the term $t$.

TF-IDF

Combining both measures provides the TF-IDF score:

$$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \text{IDF}(t, D)$$ TF-IDF assigns more weight to rare words in a document.

Cosine Similarity

Cosine similarity measures the cosine of the angle between two vectors. When used in document similarity, these vectors are typically the TF-IDF vectors representing documents. The formula for cosine similarity is:

$$\text{Cosine Similarity}(A, B) = \frac{A \cdot B}{||A|| \times ||B||}$$ Where:

• $A \cdot B$ is the dot product of vector $A$ and vector $B$.
• $||A||$ and $||B||$ are the magnitudes of vectors $A$ and $B$ respectively.

A cosine similarity of 1 indicates identical documents, while 0 indicates no similarity.

Implementing TF-IDF and Cosine Similarity in Python

We'll use the sklearn library in Python to compute TF-IDF values and to calculate cosine similarity between documents.

1from sklearn.feature_extraction.text import TfidfVectorizer
2from sklearn.metrics.pairwise import cosine_similarity
3
4# Sample documents
5documents = [
6    "The sky is blue and the sun is shining.",
7    "The weather is cold but the sky is clear.",
8    "Rain is falling and the sky is gray.",
9    "Cold days bring lots of rain."
10]
11
12# Compute TF-IDF
13tfidf_vectorizer = TfidfVectorizer()
14tfidf_matrix = tfidf_vectorizer.fit_transform(documents)
15
16# Compute Cosine Similarity
17cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)
18
19# Displaying the cosine similarity
20print(cosine_sim)

Understanding the Output

Running the above code will give us a similarity matrix. Let's consider only the results for Document 1 against others:

The diagonal values are always 1, because each document is identical to itself.

Summary

Using TF-IDF combined with cosine similarity is an effective method to measure document similarity, particularly in text mining and NLP applications. This technique helps address the challenges posed by common words, enabling more accurate analyses of document contents.

Key Points

TF-IDF and cosine similarity, combined, provide a robust framework for comparing text documents, critical for modern search algorithms and recommendation engines.