Introduction

Feature selection is the process of keeping the predictors that add useful signal and dropping the ones that mostly add noise, redundancy, or cost. In scikit-learn, feature selection is usually straightforward once you decide whether you want a simple statistical filter, a model-driven selector, or a recursive search.

Good feature selection matters for more than speed. It can improve generalization, reduce overfitting, and make a model easier to explain because you are no longer asking it to learn from every raw column.

Filter Methods for Fast Baselines

Filter methods score each feature without fitting a full predictive model repeatedly. They are fast, simple, and often a good first pass when the dataset has many obvious low-value columns.

One of the most common tools is SelectKBest. For a classification task, you can pair it with f_classif:

1from sklearn.datasets import load_breast_cancer
2from sklearn.feature_selection import SelectKBest, f_classif
3from sklearn.model_selection import train_test_split
4from sklearn.pipeline import Pipeline
5from sklearn.preprocessing import StandardScaler
6from sklearn.linear_model import LogisticRegression
7
8X, y = load_breast_cancer(return_X_y=True)
9
10X_train, X_test, y_train, y_test = train_test_split(
11    X, y, test_size=0.2, random_state=42, stratify=y
12)
13
14pipeline = Pipeline(
15    [
16        ("scale", StandardScaler()),
17        ("select", SelectKBest(score_func=f_classif, k=10)),
18        ("model", LogisticRegression(max_iter=2000)),
19    ]
20)
21
22pipeline.fit(X_train, y_train)
23print("Test accuracy:", pipeline.score(X_test, y_test))

The selector keeps the ten highest-scoring features. Putting it inside a Pipeline is important because selection should be fit on the training split only. That prevents information leakage from the test set.

Another useful baseline is VarianceThreshold, which drops columns that barely change across samples. It will not find the best predictive features, but it quickly removes dead columns such as constant flags.

Embedded Methods with Model Awareness

Embedded methods perform feature selection as part of model fitting. This is often more practical than a pure statistical filter because the model can capture how features behave in the actual prediction task.

SelectFromModel works well with sparse linear models such as L1-regularized logistic regression:

1from sklearn.feature_selection import SelectFromModel
2from sklearn.linear_model import LogisticRegression
3from sklearn.pipeline import make_pipeline
4from sklearn.preprocessing import StandardScaler
5
6selector_model = LogisticRegression(
7    penalty="l1",
8    solver="liblinear",
9    C=0.1,
10    max_iter=2000,
11)
12
13pipeline = make_pipeline(
14    StandardScaler(),
15    SelectFromModel(selector_model),
16    LogisticRegression(max_iter=2000),
17)
18
19pipeline.fit(X_train, y_train)
20print("Selected accuracy:", pipeline.score(X_test, y_test))

The first logistic-regression model shrinks weak coefficients toward zero. SelectFromModel keeps the features whose weights remain important. This is often a strong choice when you want a compact model without running an expensive subset search.

Tree-based models can also drive selection because they expose feature importances. That said, importance values from trees can be unstable when features are correlated, so treat them as a guide rather than absolute truth.

Wrapper Methods for Careful Search

Wrapper methods evaluate subsets by training a model many times. They are slower, but they can discover interactions that filter methods miss.

Recursive feature elimination is a standard example:

1from sklearn.feature_selection import RFE
2from sklearn.svm import SVC
3
4estimator = SVC(kernel="linear")
5selector = RFE(estimator=estimator, n_features_to_select=8, step=1)
6selector.fit(X_train, y_train)
7
8print("Support mask:", selector.support_)
9print("Ranking:", selector.ranking_)

This approach repeatedly removes the least useful features according to the current estimator. Because it fits the model multiple times, it can be expensive on large datasets, but it is valuable when you need a more deliberate search than SelectKBest can provide.

Choosing the Right Strategy

A practical workflow is to start with a fast filter, then compare it against one embedded method. If both produce similar performance, use the simpler pipeline. If the dataset is small and interpretability matters, try wrapper methods after you have a stable baseline.

It also helps to remember that feature selection is model-dependent. A feature set that works well for linear regression may not be best for gradient boosting. Evaluate the selector together with the model you plan to ship, not in isolation.

Common Pitfalls

• Selecting features before the train-test split. That leaks target information into evaluation and inflates the score.
• Treating feature selection as universally correct. The best subset changes with the model family and the objective.
• Ignoring correlated predictors. Some selectors keep one strong column and drop another equally valid one, which can make results look arbitrary.
• Optimizing only for the highest score. A slightly lower score with half the features may be the better engineering choice.
• Forgetting preprocessing in the pipeline. Scaling should usually happen before selection for distance-based and coefficient-based methods.

Summary

• Scikit-learn supports filter, embedded, and wrapper feature-selection strategies.
• 'SelectKBest is a strong baseline when you need speed and simplicity.'
• 'SelectFromModel uses model behavior directly and is often a practical middle ground.'
• Wrapper methods such as RFE are slower but can find better subsets on smaller problems.
• Put selection inside a Pipeline so it is fit only on training data.