Introduction

Fisher's Linear Discriminant is a supervised dimensionality-reduction method that projects data onto a direction designed to separate classes as much as possible. In Python, you can implement the two-class version directly with NumPy or use a library implementation such as scikit-learn's LDA when you want a production-ready tool.

The Core Idea

For two classes, Fisher's method looks for a projection vector w that makes class means far apart after projection while keeping samples within each class tightly clustered.

The classic result is:

w = S_w^(-1) (mu1 - mu2)

where:

• 'mu1 and mu2 are the class means,'
• and S_w is the within-class scatter matrix.

The projection does not try to preserve raw variance the way PCA does. It tries to preserve class separation.

Build the Scatter Matrices

Suppose X1 and X2 are the samples from two classes. Then:

• compute the mean of each class,
• center each class around its own mean,
• accumulate the within-class scatter,
• and solve for the projection direction.

Here is a minimal NumPy implementation:

1import numpy as np
2
3def fisher_linear_discriminant(X1, X2):
4    mu1 = X1.mean(axis=0)
5    mu2 = X2.mean(axis=0)
6
7    S1 = (X1 - mu1).T @ (X1 - mu1)
8    S2 = (X2 - mu2).T @ (X2 - mu2)
9    Sw = S1 + S2
10
11    w = np.linalg.solve(Sw, mu1 - mu2)
12    return w
13
14
15X1 = np.array([
16    [2.0, 3.0],
17    [3.0, 3.5],
18    [2.5, 2.8],
19])
20
21X2 = np.array([
22    [6.0, 5.0],
23    [7.0, 5.5],
24    [6.5, 4.8],
25])
26
27w = fisher_linear_discriminant(X1, X2)
28print(w)

Using np.linalg.solve is usually better than explicitly computing a matrix inverse.

Project the Data

Once you have w, project each sample onto that line:

1def project(X, w):
2    return X @ w
3
4proj1 = project(X1, w)
5proj2 = project(X2, w)
6
7print("Class 1 projections:", proj1)
8print("Class 2 projections:", proj2)

If the classes are well separated in the original space, their projected values should form distinct ranges or at least become easier to classify with a threshold.

Turn It Into a Simple Classifier

For a two-class toy example, one simple classifier uses the midpoint between projected class means as a threshold.

1mean1 = proj1.mean()
2mean2 = proj2.mean()
3threshold = (mean1 + mean2) / 2
4
5def predict(X, w, threshold):
6    projected = X @ w
7    if mean1 < mean2:
8        return (projected > threshold).astype(int)
9    return (projected < threshold).astype(int)
10
11X_test = np.array([
12    [2.7, 3.1],
13    [6.2, 5.1],
14])
15
16print(predict(X_test, w, threshold))

This is not the most sophisticated classifier, but it makes Fisher's method concrete: first find a discriminative direction, then classify based on projected position.

Relationship to LDA

In practice, Fisher's Linear Discriminant is closely related to Linear Discriminant Analysis. Scikit-learn wraps the broader method in a familiar API:

1from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
2import numpy as np
3
4X = np.vstack([X1, X2])
5y = np.array([0] * len(X1) + [1] * len(X2))
6
7model = LinearDiscriminantAnalysis(n_components=1)
8model.fit(X, y)
9
10print(model.transform(X))

If your goal is practical classification, this is usually the best route. If your goal is understanding the method, the NumPy version is more instructive.

Numerical Issues Matter

If the within-class scatter matrix is singular or poorly conditioned, the direct solve may become unstable. That happens when:

• features are highly collinear,
• there are more features than samples,
• or one feature is a linear combination of others.

Common fixes include regularization, dimensionality reduction before LDA, or using a library implementation that already handles these cases more carefully.

Fisher Versus PCA

This comparison matters because the two are often confused:

• PCA ignores labels and preserves variance
• Fisher's method uses labels and preserves class separation

A direction with large variance is not always the direction that best separates two classes. That is exactly why Fisher's method exists.

Common Pitfalls

The biggest pitfall is applying Fisher's method as if it were just another unsupervised projection technique. It needs class labels and uses them directly.

Another mistake is explicitly inverting S_w with np.linalg.inv instead of solving the system. Direct inversion is often less numerically stable.

Developers also sometimes expect perfect separation even when the classes overlap substantially. Fisher's method finds the best linear projection under the model, not a magic separator for arbitrary data.

Finally, if the feature count is large relative to the sample count, watch for singular matrices and consider regularization or library implementations.

Summary

• Fisher's Linear Discriminant finds a projection that maximizes class separation relative to within-class spread.
• For two classes, the core computation is w = S_w^(-1) (mu1 - mu2).
• A NumPy implementation is straightforward and useful for learning the method.
• Scikit-learn's LinearDiscriminantAnalysis is the practical choice for real projects.
• Numerical stability and class overlap are the main practical issues to watch.