Introduction

There is no single correct order for feature selection, PCA, and normalization in every machine learning problem, but there is a reliable principle: split the data first, then fit every preprocessing step only on the training portion. After that, the typical numeric pipeline is scaling before PCA, with feature selection placed according to what kind of selector you are using and why you are using it.

Start with the Leak-Free Pipeline Order

Before debating PCA versus feature selection, get the leakage order right.

1. split train and test data
2. fit preprocessing only on the training data
3. apply the learned transformations to validation and test data
4. evaluate the final pipeline on untouched data

This matters because scaling, feature selection, and PCA all learn something from the data. If you fit them before the train-test split, information from the test set leaks into training.

In scikit-learn, a Pipeline is the safest way to enforce the order.

1from sklearn.pipeline import Pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.decomposition import PCA
4from sklearn.linear_model import LogisticRegression
5
6pipeline = Pipeline([
7    ("scale", StandardScaler()),
8    ("pca", PCA(n_components=5)),
9    ("model", LogisticRegression(max_iter=1000)),
10])

Scale Before PCA

PCA is based on variance. If one feature has much larger numeric scale than another, it can dominate the principal components even when it is not intrinsically more informative.

That is why the usual order for numeric inputs is:

• clean or impute data
• scale numeric features
• run PCA

1from sklearn.preprocessing import StandardScaler
2from sklearn.decomposition import PCA
3
4X_scaled = StandardScaler().fit_transform(X_train)
5X_reduced = PCA(n_components=10).fit_transform(X_scaled)

If your data is already on comparable scales and the model goal justifies it, scaling may be less critical, but for many tabular problems standardization before PCA is the default safe choice.

Feature Selection and PCA Solve Different Problems

Feature selection chooses a subset of original features. PCA creates new synthetic features that are linear combinations of the originals.

That difference matters:

• feature selection helps interpretability
• PCA helps compression and decorrelation
• using both is not automatically better

If you need to preserve original feature meaning, start with feature selection. If you mainly want dimension reduction for modeling efficiency or collinearity control, PCA may be the better tool.

Where Feature Selection Belongs

The placement of feature selection depends on the selector type.

For variance-based or distance-sensitive selectors, scale first.

For univariate supervised selectors, the common order is:

• scale if needed by the selector or downstream model
• fit the selector on training data
• optionally apply PCA after selection if the remaining dimension is still high

1from sklearn.feature_selection import SelectKBest, f_classif
2from sklearn.pipeline import Pipeline
3from sklearn.preprocessing import StandardScaler
4from sklearn.linear_model import LogisticRegression
5
6pipeline = Pipeline([
7    ("scale", StandardScaler()),
8    ("select", SelectKBest(score_func=f_classif, k=20)),
9    ("model", LogisticRegression(max_iter=1000)),
10])

In many cases, you should compare two pipelines instead of forcing both steps into one:

• scaling plus feature selection plus model
• scaling plus PCA plus model

Tune the Whole Pipeline with Cross-Validation

The right order is not just a conceptual issue; it is a model-selection issue. If you are unsure whether PCA helps, compare it with feature selection under cross-validation.

That gives you an evidence-based answer instead of a rule-of-thumb answer.

Common Pitfalls

• Fitting scaling, PCA, or feature selection before the train-test split and causing data leakage.
• Applying PCA before scaling on numeric features with very different ranges.
• Using both feature selection and PCA automatically without a clear reason.
• Expecting PCA to preserve the interpretability of original features.
• Tuning preprocessing steps outside cross-validation instead of inside a full pipeline.

Summary

• Split data first, then fit every preprocessing step on training data only.
• For numeric data, scaling usually comes before PCA.
• Feature selection and PCA solve different problems and should not be combined by reflex.
• Put selectors where their assumptions make sense, often after scaling.
• Compare full pipelines with cross-validation instead of relying on one fixed recipe.