Introduction

Support Vector Machines can handle high-dimensional data well, but that does not mean every feature helps. Irrelevant or redundant predictors can slow training, reduce interpretability, and sometimes hurt generalization. In R, a practical SVM feature-selection workflow usually combines a clean train-test split, optional filter steps, and a wrapper method such as recursive feature elimination.

Start with a Baseline Model

Before selecting features, fit a simple baseline SVM so you have something to compare against later.

1set.seed(42)
2
3library(caret)
4library(e1071)
5
6data(iris)
7
8idx <- createDataPartition(iris$Species, p = 0.8, list = FALSE)
9train_df <- iris[idx, ]
10test_df <- iris[-idx, ]
11
12baseline_model <- svm(
13  Species ~ .,
14  data = train_df,
15  kernel = "radial",
16  gamma = 0.1,
17  cost = 1
18)
19
20baseline_pred <- predict(baseline_model, test_df)
21confusionMatrix(baseline_pred, test_df$Species)

This does two things. First, it tells you whether the current feature set already performs well. Second, it stops feature selection from becoming a ritual with no measurable benefit.

Use Simple Filters to Remove Obvious Noise

A fast filter stage can remove features that are clearly problematic before you run a wrapper method. Common checks include near-zero variance and high correlation.

1x <- train_df[, -5]
2y <- train_df$Species
3
4nzv <- nearZeroVar(x)
5if (length(nzv) > 0) {
6  x <- x[, -nzv, drop = FALSE]
7}
8
9corr_matrix <- cor(x)
10high_corr <- findCorrelation(corr_matrix, cutoff = 0.9)
11if (length(high_corr) > 0) {
12  x <- x[, -high_corr, drop = FALSE]
13}
14
15filtered_train <- data.frame(x, Species = y)
16names(filtered_train)

These filters are not SVM-specific, but they reduce the search space and often remove obviously weak predictors early.

Apply Recursive Feature Elimination

Wrapper methods score candidate subsets using model performance. In caret, rfe is a practical tool for this.

1set.seed(42)
2
3ctrl <- rfeControl(
4  functions = caretFuncs,
5  method = "repeatedcv",
6  number = 5,
7  repeats = 2
8)
9
10rfe_fit <- rfe(
11  x = filtered_train[, setdiff(names(filtered_train), "Species")],
12  y = filtered_train$Species,
13  sizes = 1:4,
14  rfeControl = ctrl,
15  method = "svmRadial",
16  trControl = trainControl(method = "cv", number = 5)
17)
18
19print(rfe_fit)
20best_vars <- predictors(rfe_fit)
21print(best_vars)

rfe repeatedly trains and scores models with different feature subsets. The best subset is chosen from resampling results rather than from a single lucky split.

Retrain the Final SVM on the Selected Features

After feature selection, retrain the model using only the chosen predictors and evaluate it on the untouched test set.

1formula_txt <- paste("Species ~", paste(best_vars, collapse = " + "))
2
3final_model <- svm(
4  as.formula(formula_txt),
5  data = train_df,
6  kernel = "radial",
7  gamma = 0.1,
8  cost = 1
9)
10
11final_pred <- predict(final_model, test_df)
12confusionMatrix(final_pred, test_df$Species)

At this point, compare the final metrics with the baseline metrics. If performance is unchanged, the selected subset may still be valuable because it simplifies the model. If performance drops, your feature-selection method may be too aggressive.

Tune Hyperparameters with the Selected Features

Feature choice and SVM hyperparameters interact. A subset that works well for one cost and gamma pair may not be best for another. Once you have a promising feature set, tune the SVM using those features.

1set.seed(42)
2
3grid <- expand.grid(
4  sigma = c(0.01, 0.05, 0.1),
5  C = c(0.5, 1, 2, 5)
6)
7
8tuned_model <- train(
9  as.formula(formula_txt),
10  data = train_df,
11  method = "svmRadial",
12  trControl = trainControl(method = "repeatedcv", number = 5, repeats = 2),
13  tuneGrid = grid
14)
15
16print(tuned_model$bestTune)

This step matters because otherwise you may judge a feature subset using poor hyperparameter defaults.

Keep the Workflow Reproducible

Save the selected feature names along with the trained model so later predictions use the same schema.

saveRDS(final_model, "svm_model.rds")
writeLines(best_vars, con = "selected_features.txt")

That small habit prevents subtle deployment bugs where new data arrives with extra or reordered columns.

Common Pitfalls

The biggest mistake is doing feature selection on the full dataset before the train-test split. That leaks information from the test set and produces overly optimistic results.

Another issue is assuming the selected features are universally best. Change the kernel, scaling, or tuning grid, and the useful subset may change too.

It is also common to skip a baseline comparison. If you never compare against the all-feature model, you cannot tell whether feature selection helped or just added complexity.

Finally, remember that SVMs are sensitive to preprocessing. Keep scaling and selection decisions inside a controlled workflow.

Summary

• Fit a baseline SVM before selecting features so you have a fair comparison.
• Use fast filters to remove obvious low-value predictors.
• Apply caret::rfe or a similar wrapper method to score subsets with resampling.
• Retrain and tune the final SVM using the selected variables only.
• Avoid leakage by keeping selection inside the training workflow.