Multivariate Outlier Removal With Mahalanobis Distance

Multivariate Analysis

Outlier Detection

Mahalanobis Distance

Statistical Methods

Data Preprocessing

Multivariate Outlier Removal With Mahalanobis Distance

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

Mahalanobis distance measures how far a point is from the center of a multivariate distribution, accounting for correlations between variables. Unlike Euclidean distance, which treats all directions equally, Mahalanobis distance stretches or compresses axes based on the covariance structure of the data. Points with a large Mahalanobis distance are multivariate outliers — they may look normal in each variable individually but are unusual when considering the relationships between variables.

The Formula

The Mahalanobis distance of a point x from a distribution with mean mu and covariance matrix S is:

D_M(x) = sqrt( (x - mu)^T * S^(-1) * (x - mu) )

x is the observation vector
mu is the mean vector of the dataset
S is the covariance matrix
S^(-1) is the inverse covariance matrix (precision matrix)

The key insight: multiplying by the inverse covariance matrix decorrelates the variables and normalizes their scales, so variables with high variance do not dominate the distance calculation.

Python Implementation

python

1import numpy as np
2from scipy.spatial.distance import mahalanobis
3from scipy.stats import chi2
4
5# Generate sample data with an outlier
6np.random.seed(42)
7data = np.random.multivariate_normal(
8    mean=[0, 0],
9    cov=[[1, 0.8], [0.8, 1]],
10    size=100
11)
12# Add an outlier
13data = np.vstack([data, [4, -3]])
14
15# Compute Mahalanobis distance for each point
16mean = np.mean(data, axis=0)
17cov = np.cov(data, rowvar=False)
18cov_inv = np.linalg.inv(cov)
19
20distances = np.array([
21    mahalanobis(point, mean, cov_inv) for point in data
22])
23
24print(f"Outlier distance: {distances[-1]:.2f}")  # Much larger than others
25print(f"Mean distance: {distances[:-1].mean():.2f}")

Setting the Threshold with Chi-Square

For multivariate normal data, the squared Mahalanobis distance follows a chi-square distribution with p degrees of freedom (where p is the number of variables):

python

1from scipy.stats import chi2
2
3p = data.shape[1]  # Number of variables
4alpha = 0.01       # Significance level (1% false positive rate)
5
6threshold = chi2.ppf(1 - alpha, df=p)
7print(f"Threshold (alpha={alpha}): {threshold:.2f}")
8
9# Flag outliers
10d_squared = distances ** 2
11outlier_mask = d_squared > threshold
12n_outliers = outlier_mask.sum()
13print(f"Outliers detected: {n_outliers}")
14
15# Remove outliers
16clean_data = data[~outlier_mask]

Common alpha values:

alpha=0.05 (95% confidence) — moderate, catches more outliers
alpha=0.01 (99% confidence) — conservative, fewer false positives
alpha=0.001 (99.9% confidence) — very conservative

Complete Outlier Removal Function

python

1import numpy as np
2from scipy.spatial.distance import mahalanobis
3from scipy.stats import chi2
4
5def remove_mahalanobis_outliers(data, alpha=0.01):
6    """Remove multivariate outliers using Mahalanobis distance.
7
8    Args:
9        data: numpy array of shape (n_samples, n_features)
10        alpha: significance level for chi-square threshold
11
12    Returns:
13        clean_data: array with outliers removed
14        outlier_mask: boolean array (True = outlier)
15    """
16    mean = np.mean(data, axis=0)
17    cov = np.cov(data, rowvar=False)
18    cov_inv = np.linalg.inv(cov)
19
20    distances_sq = np.array([
21        mahalanobis(x, mean, cov_inv) ** 2 for x in data
22    ])
23
24    threshold = chi2.ppf(1 - alpha, df=data.shape[1])
25    outlier_mask = distances_sq > threshold
26
27    return data[~outlier_mask], outlier_mask
28
29# Usage
30clean_data, mask = remove_mahalanobis_outliers(data, alpha=0.01)
31print(f"Removed {mask.sum()} outliers from {len(data)} points")

With pandas DataFrames

python

1import pandas as pd
2import numpy as np
3from scipy.spatial.distance import mahalanobis
4from scipy.stats import chi2
5
6df = pd.DataFrame({
7    'height': [170, 175, 168, 180, 165, 210, 172],
8    'weight': [70, 80, 65, 85, 60, 55, 75],
9    'age': [30, 35, 28, 40, 25, 32, 33]
10})
11
12# Select numeric columns
13numeric_cols = ['height', 'weight', 'age']
14data = df[numeric_cols].values
15
16mean = data.mean(axis=0)
17cov = np.cov(data, rowvar=False)
18cov_inv = np.linalg.inv(cov)
19
20df['mahal_dist'] = [mahalanobis(x, mean, cov_inv) for x in data]
21df['mahal_dist_sq'] = df['mahal_dist'] ** 2
22
23threshold = chi2.ppf(0.99, df=len(numeric_cols))
24df['is_outlier'] = df['mahal_dist_sq'] > threshold
25
26print(df[['height', 'weight', 'age', 'mahal_dist', 'is_outlier']])
27# The person with height=210, weight=55 is likely flagged as outlier
28# (tall but unusually light — unusual combination)
29
30clean_df = df[~df['is_outlier']].drop(columns=['mahal_dist', 'mahal_dist_sq', 'is_outlier'])

Scikit-Learn Approach

python

1from sklearn.covariance import EllipticEnvelope
2
3# EllipticEnvelope uses Mahalanobis distance internally
4detector = EllipticEnvelope(contamination=0.05)  # Expect 5% outliers
5detector.fit(data)
6
7predictions = detector.predict(data)  # 1 = inlier, -1 = outlier
8clean_data = data[predictions == 1]
9print(f"Removed {(predictions == -1).sum()} outliers")

EllipticEnvelope uses the Minimum Covariance Determinant (MCD) estimator, which is more robust to outliers than the standard covariance matrix.

Why Not Just Use Euclidean Distance?

python

1# Example: height (cm) and weight (kg) are correlated
2# A person who is 190cm and 90kg is normal
3# A person who is 190cm and 50kg is unusual
4
5# Euclidean distance treats both as equally far from the mean
6# Mahalanobis distance accounts for the height-weight correlation
7# and correctly identifies the 190cm/50kg person as an outlier

Euclidean distance also fails when variables have different scales (e.g., height in cm vs weight in kg). Mahalanobis distance normalizes scales automatically through the covariance matrix.

Common Pitfalls

Singular covariance matrix: When features are perfectly correlated or there are more features than samples, the covariance matrix is singular and cannot be inverted. Use np.linalg.pinv() (pseudo-inverse) or reduce dimensions with PCA first.
Non-normal data: Mahalanobis distance assumes multivariate normality. For skewed or heavy-tailed data, the chi-square threshold produces unreliable results. Transform data (log, Box-Cox) or use robust methods like EllipticEnvelope.
Outliers contaminating the covariance estimate: The standard covariance is itself affected by outliers, making the distance less reliable. Use the Minimum Covariance Determinant (MCD) estimator from sklearn.covariance.MinCovDet for a robust covariance.
High dimensionality: In high dimensions, Mahalanobis distance requires many more samples than features for a stable covariance estimate. As a rule of thumb, you need at least 5-10x more samples than features.
Removing too many points: With alpha=0.05, you expect to flag 5% of normal data as outliers even when there are none. Choose alpha conservatively and inspect flagged points before removing them.

Summary

Mahalanobis distance measures how unusual a point is considering correlations between variables
Squared Mahalanobis distance follows a chi-square distribution — use it to set a statistical threshold
Use scipy.spatial.distance.mahalanobis for computation and scipy.stats.chi2.ppf for the threshold
Use sklearn.covariance.EllipticEnvelope for a robust, ready-made solution
Always check for singular covariance matrices and non-normal data before applying
Start with alpha=0.01 and inspect flagged points rather than blindly removing them