Introduction

When you train a KNN classifier with caret, the default tuning behavior usually optimizes accuracy. That is often the wrong target for imbalanced classification, where a model can get many cases right simply by favoring the majority class. If you care about the balance between precision and recall, you need to teach caret how to calculate F1 and then tell train() to optimize that value.

Why F1 Needs a Custom Summary

caret::train() chooses the best hyperparameters from the metrics returned by the resampling summary function. For classification, the built-in summaries usually return values such as accuracy, kappa, ROC, sensitivity, or specificity depending on the control settings. F1 is not automatically used just because you set metric = "F1".

That means two things must line up:

1. trainControl() needs a summaryFunction that returns a named value called F1.
2. train() must set metric = "F1" so the tuning loop picks the row with the best F1 score.

If either part is missing, caret will not optimize what you think it is optimizing.

Define a Binary F1 Summary Function

For binary classification, the simplest approach is to compute precision and recall from the resampled predictions and combine them into the harmonic mean.

1library(caret)
2
3f1_summary <- function(data, lev = NULL, model = NULL) {
4  positive_class <- lev[1]
5
6  precision <- posPredValue(data$pred, data$obs, positive = positive_class)
7  recall <- sensitivity(data$pred, data$obs, positive = positive_class)
8
9  if (is.na(precision) || is.na(recall) || precision + recall == 0) {
10    f1 <- 0
11  } else {
12    f1 <- 2 * precision * recall / (precision + recall)
13  }
14
15  c(F1 = f1)
16}

The returned object must be a named numeric vector. The name matters because metric = "F1" refers to that label exactly.

Train KNN With F1 as the Selection Metric

A complete training example should also include scaling, because KNN is sensitive to feature magnitude. The following example uses the twoClassSim() helper from caret so the code is directly runnable.

1library(caret)
2
3set.seed(42)
4train_data <- twoClassSim(300)
5
6train_data$Class <- factor(train_data$Class, levels = c("Class1", "Class2"))
7
8ctrl <- trainControl(
9  method = "cv",
10  number = 5,
11  summaryFunction = f1_summary,
12  classProbs = FALSE,
13  savePredictions = "final"
14)
15
16knn_model <- train(
17  Class ~ .,
18  data = train_data,
19  method = "knn",
20  preProcess = c("center", "scale"),
21  tuneLength = 10,
22  trControl = ctrl,
23  metric = "F1"
24)
25
26knn_model

This tells caret to test several k values and select the one with the highest cross-validated F1 score.

Be Explicit About the Positive Class

F1 is not symmetric in the way many people assume. The class treated as “positive” determines which precision and recall values are used. In caret, the positive class often follows the order of the factor levels.

If your business problem is “detect fraud,” the fraud label should be the positive class. Reversing the factor order can produce an F1 score for the wrong event, which makes the reported metric technically valid but operationally misleading.

fraud_data$target <- factor(fraud_data$target, levels = c("fraud", "not_fraud"))

Make that explicit before training rather than trusting the default level ordering from the source data.

Validate Beyond One Metric

F1 is useful, but it should not replace inspection of the confusion matrix. Two models can have similar F1 values while failing in different ways. One may have poor recall. Another may have poor precision. The business cost of those errors may not be the same.

pred <- predict(knn_model, newdata = train_data)
confusionMatrix(pred, train_data$Class, positive = "Class1")

This lets you inspect the class-level behavior directly. It is also a good way to catch problems caused by a mismatched positive class or weak preprocessing.

Multi-Class F1 Requires a Different Design

If the problem has more than two classes, saying “use F1” is incomplete. You need to define whether you want macro F1, micro F1, weighted F1, or one-vs-rest F1 for a single class. caret will not choose that interpretation for you.

For many teams, the best path is to keep the first version simple:

1. Start with a binary framing if the business question naturally has one critical class.
2. Use a custom summary that returns the exact F1 variant you need.
3. Document that metric so the tuning objective is not ambiguous.

That extra clarity matters more than the metric label itself.

Common Pitfalls

• Setting metric = "F1" without defining a summary function that actually returns a value named F1.
• Letting the factor levels define the wrong positive class for the problem you care about.
• Training KNN on unscaled numeric features and then blaming the metric for poor results.
• Assuming binary F1 logic applies unchanged to a multi-class problem.
• Looking only at the best F1 value instead of inspecting the confusion matrix and error tradeoffs.

Summary

• 'caret can optimize F1 for KNN, but only if you provide a custom summary function.'
• The summary function must return a named metric that matches metric = "F1".
• Feature scaling is still essential because KNN depends on distances.
• The positive class must be chosen deliberately or the score can be misleading.
• F1 is a strong tuning target for imbalanced binary classification when precision and recall both matter.