Introduction

Recursive feature elimination, or RFE, repeatedly fits a model, ranks features, removes the weakest ones, and evaluates the smaller feature sets. In R, a practical way to do this with an SVM is to use caret::rfe together with a linear SVM model so the ranking step is well defined.

Why Linear SVM Is the Usual Starting Point

RFE needs some notion of feature importance so it can decide what to remove next. With SVMs, that is simplest when you use a linear kernel, because the model has coefficients that can support ranking.

That is why most RFE-with-SVM examples start with a linear SVM rather than a non-linear kernel.

A Runnable caret Example

The caret package provides an rfe function that can repeatedly train and evaluate candidate feature subsets.

1library(caret)
2
3set.seed(42)
4data(iris)
5
6x <- iris[, 1:4]
7y <- iris$Species
8
9ctrl <- rfeControl(
10  functions = caretFuncs,
11  method = "cv",
12  number = 5
13)
14
15sizes <- c(1, 2, 3, 4)
16
17svm_profile <- rfe(
18  x = x,
19  y = y,
20  sizes = sizes,
21  rfeControl = ctrl,
22  method = "svmLinear",
23  preProcess = c("center", "scale"),
24  trControl = trainControl(method = "cv")
25)
26
27print(svm_profile)
28predictors(svm_profile)
29plot(svm_profile, type = c("g", "o"))

This code does several important things:

• uses cross-validation during the feature-selection process
• evaluates several candidate subset sizes
• trains a linear SVM for each subset
• reports the chosen feature set and its resampling performance

How to Read the Result

The print output summarizes performance for each subset size and reports the selected number of predictors. predictors(svm_profile) returns the final chosen feature names.

The plot is also useful because it shows whether model performance improves, plateaus, or degrades as features are removed.

That matters because RFE is not about finding the smallest possible set. It is about finding a smaller set that still predicts well.

Training a Final Model on the Selected Features

Once RFE chooses a subset, you can train a final SVM using only those predictors.

1selected <- predictors(svm_profile)
2
3final_model <- train(
4  x = x[, selected, drop = FALSE],
5  y = y,
6  method = "svmLinear",
7  preProcess = c("center", "scale"),
8  trControl = trainControl(method = "cv")
9)
10
11print(final_model)

This makes the modeling pipeline explicit: first select features, then train the final model on that selected subset.

Why Preprocessing Matters

SVMs are sensitive to feature scale, so centering and scaling are usually important. In caret, pass preProcess = c("center", "scale") so the resampling workflow applies preprocessing consistently.

Do not scale the full dataset outside the resampling process and then run RFE on the already-transformed data. That can leak information across folds.

What About Non-Linear SVMs

You can fit non-linear SVMs in R, but feature ranking becomes less straightforward because there is no simple coefficient vector analogous to the linear case.

That does not make non-linear models bad. It just means linear SVM is the natural starting point when the goal is coefficient-driven recursive elimination.

Common Pitfalls

A common mistake is using RFE with an SVM method that does not provide a clean ranking signal and then assuming the selected features are trustworthy.

Another mistake is forgetting to scale predictors. SVMs can behave poorly when one feature dominates only because of its numeric range.

A third issue is interpreting the chosen feature subset as universally optimal. RFE results depend on the data, resampling strategy, and model settings.

Summary

• In R, caret::rfe is a practical way to run recursive feature elimination with SVM
• A linear SVM is the usual choice because feature ranking is clearer
• Use cross-validation inside the RFE workflow, not just after it
• Center and scale predictors during the resampling process
• After selection, train a final model using the chosen feature subset