Introduction

Principal Component Analysis, or PCA, is a linear dimensionality-reduction technique that projects data onto new axes chosen to capture as much variance as possible. It is popular because it can compress high-dimensional data, reduce noise, and make visualization easier without requiring labels.

The Core Idea

PCA does not choose arbitrary new features. It finds orthogonal directions in the data, called principal components, ordered by how much variance they explain.

The first principal component explains the largest possible variance. The second explains the largest remaining variance subject to being orthogonal to the first, and so on.

That means PCA answers this question:

• if I can keep only a few directions, which ones preserve the most information about how the data varies

Why Centering Matters

Before PCA, the data is usually centered by subtracting the mean of each feature.

1import numpy as np
2
3X = np.array([
4    [2.0, 1.0],
5    [3.0, 2.0],
6    [4.0, 3.0],
7])
8
9X_centered = X - X.mean(axis=0)
10print(X_centered)

Without centering, the first component can be pulled toward the mean offset instead of capturing the true variance structure.

SVD Is the Practical Engine

Although PCA is often explained through the covariance matrix and eigenvectors, many implementations use singular value decomposition internally because it is numerically stable.

In scikit-learn, you can use PCA directly:

1from sklearn.decomposition import PCA
2from sklearn.preprocessing import StandardScaler
3import numpy as np
4
5X = np.array([
6    [1.0, 2.0, 3.0],
7    [2.0, 3.0, 4.0],
8    [3.0, 4.0, 5.0],
9    [4.0, 5.0, 6.0],
10])
11
12X_scaled = StandardScaler().fit_transform(X)
13pca = PCA(n_components=2)
14X_reduced = pca.fit_transform(X_scaled)
15
16print(X_reduced)
17print(pca.explained_variance_ratio_)

This reduces a 3-feature dataset to 2 principal components while showing how much variance each retained component explains.

Choosing the Number of Components

One common strategy is to keep enough components to explain a target fraction of variance.

pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X_scaled)
print(X_reduced.shape)

Here scikit-learn chooses the smallest number of components that preserve about 95 percent of the variance.

This is convenient when you care more about retained information than about a fixed output dimension.

What PCA Is Good For

PCA is especially useful for:

• reducing wide feature spaces before modeling
• visualizing data in 2D or 3D
• removing redundant linear correlations
• denoising when small-variance directions mostly contain noise

A classic example is projecting image or gene-expression data into a smaller space before clustering or classification.

What PCA Does Not Guarantee

PCA does not know anything about the target label. That means high-variance directions are not always the most predictive directions for a supervised task.

It is also linear. If the data lies on a curved manifold, PCA may miss the underlying low-dimensional structure.

That is why PCA is powerful but not universal.

Standardization Often Matters

If your features are on different scales, PCA can be dominated by the largest-scale feature.

For example, if one feature ranges from 0 to 1 and another from 0 to 100000, the second feature can overwhelm the variance calculation.

That is why standardization is often done before PCA, especially when features use different units.

Interpreting the Components

Each principal component is a weighted combination of the original features.

You can inspect the loadings:

print(pca.components_)

Large positive or negative weights show which original features contribute strongly to a component.

Interpretation can still be hard, though, because each component mixes multiple features rather than preserving one feature name directly.

Common Pitfalls

A common mistake is applying PCA without centering or scaling data appropriately.

Another issue is keeping components solely because they are easy to visualize rather than because they preserve the right amount of information for the task.

Developers also sometimes assume PCA will always improve a classifier. It can help, but it can also remove signal if the discarded low-variance dimensions were actually predictive.

Finally, do not interpret principal components as causal factors. They are variance-maximizing directions, not discovered causes.

Summary

• PCA reduces dimensionality by projecting data onto variance-maximizing orthogonal directions.
• Centering and often scaling are important preprocessing steps.
• Practical implementations usually rely on SVD.
• PCA is useful for compression, denoising, and visualization.
• It is linear, unsupervised, and not guaranteed to preserve the most predictive features.