Introduction

By default, scikit-learn's KNeighborsClassifier uses Euclidean distance (Minkowski with p=2) to find nearest neighbors. For text classification, recommendation systems, and other high-dimensional sparse data problems, cosine distance often produces better results because it measures the angle between vectors rather than their magnitude. This article explains how to configure KNeighborsClassifier to use cosine distance.

Cosine Similarity vs Cosine Distance

Cosine similarity measures how similar two vectors are by computing the cosine of the angle between them:

cosine_similarity(A, B) = (A · B) / (||A|| * ||B||)

The result ranges from -1 (opposite) to 1 (identical direction). For non-negative vectors (like TF-IDF), it ranges from 0 to 1.

Cosine distance is simply:

cosine_distance(A, B) = 1 - cosine_similarity(A, B)

This converts similarity to a distance metric where 0 means identical and values closer to 1 (or 2) mean more different.

Using Cosine Distance in KNeighborsClassifier

Method 1: Using the metric Parameter

Pass metric='cosine' directly to the classifier:

1from sklearn.neighbors import KNeighborsClassifier
2
3knn = KNeighborsClassifier(n_neighbors=5, metric='cosine')
4knn.fit(X_train, y_train)
5predictions = knn.predict(X_test)

This uses scikit-learn's built-in cosine distance implementation.

Method 2: Using a Custom Distance Function

For more control, define a custom distance function:

1from sklearn.neighbors import KNeighborsClassifier
2from sklearn.metrics.pairwise import cosine_distances
3import numpy as np
4
5def custom_cosine(x, y):
6    dot = np.dot(x, y)
7    norm_x = np.linalg.norm(x)
8    norm_y = np.linalg.norm(y)
9    if norm_x == 0 or norm_y == 0:
10        return 1.0  # Maximum distance for zero vectors
11    return 1.0 - dot / (norm_x * norm_y)
12
13knn = KNeighborsClassifier(n_neighbors=5, metric=custom_cosine)
14knn.fit(X_train, y_train)
15predictions = knn.predict(X_test)

Method 3: Pre-normalize and Use Euclidean Distance

A computationally efficient alternative is to L2-normalize your vectors first. When vectors are unit length, Euclidean distance and cosine distance produce the same ranking:

1from sklearn.preprocessing import normalize
2from sklearn.neighbors import KNeighborsClassifier
3
4# L2-normalize the features
5X_train_norm = normalize(X_train, norm='l2')
6X_test_norm = normalize(X_test, norm='l2')
7
8# Euclidean distance on normalized vectors ≈ cosine distance
9knn = KNeighborsClassifier(n_neighbors=5, metric='euclidean')
10knn.fit(X_train_norm, y_train)
11predictions = knn.predict(X_test_norm)

This approach is faster because the ball_tree and kd_tree algorithms (not available for cosine) can be used with Euclidean distance.

Complete Example: Text Classification

1from sklearn.datasets import fetch_20newsgroups
2from sklearn.feature_extraction.text import TfidfVectorizer
3from sklearn.neighbors import KNeighborsClassifier
4from sklearn.metrics import classification_report
5from sklearn.model_selection import train_test_split
6
7# Load text data
8categories = ['sci.space', 'comp.graphics', 'rec.sport.baseball']
9newsgroups = fetch_20newsgroups(subset='all', categories=categories)
10
11# Convert text to TF-IDF vectors
12vectorizer = TfidfVectorizer(max_features=5000, stop_words='english')
13X = vectorizer.fit_transform(newsgroups.data)
14y = newsgroups.target
15
16# Split data
17X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
18
19# KNN with cosine distance
20knn_cosine = KNeighborsClassifier(n_neighbors=5, metric='cosine', algorithm='brute')
21knn_cosine.fit(X_train, y_train)
22
23# Evaluate
24y_pred = knn_cosine.predict(X_test)
25print(classification_report(y_test, y_pred, target_names=categories))

Algorithm Compatibility

Not all KNN algorithms support cosine distance:

When using metric='cosine', set algorithm='brute' explicitly to avoid warnings:

knn = KNeighborsClassifier(n_neighbors=5, metric='cosine', algorithm='brute')

Common Pitfalls

• Sparse matrix support: metric='cosine' with algorithm='brute' works with sparse matrices (like TF-IDF output). Custom distance functions may not handle sparse input — convert to dense first or handle explicitly.
• Zero vectors: Cosine distance is undefined for zero vectors. Scikit-learn handles this gracefully, but custom functions should check for zero norms.
• Performance with large datasets: The brute algorithm has O(n*d) time complexity per query, which is slow for large datasets. For better performance, use the normalization trick with kd_tree or approximate nearest neighbor libraries like FAISS or Annoy.
• Choosing k: With cosine distance in high-dimensional spaces, the optimal k is often smaller than with Euclidean distance. Use cross-validation to tune it.
• Negative values: Cosine similarity can range from -1 to 1, making cosine distance range from 0 to 2. Some implementations assume non-negative features. Verify behavior with your data.

Summary

• Use metric='cosine' in KNeighborsClassifier for angle-based similarity
• Set algorithm='brute' since tree-based algorithms do not support cosine
• For better performance, L2-normalize features and use Euclidean distance instead
• Cosine distance is especially effective for text (TF-IDF), recommendation, and high-dimensional sparse data