Introduction

If caret::train looks like it is still optimizing ROC after you asked for sensitivity, the usual issue is not that caret is ignoring your setting. The real cause is usually one of three things: the metric name does not match the summary function output, the positive class is not the class you think it is, or sensitivity and ROC move together for the chosen model grid.

What caret Actually Optimizes

caret optimizes the column named by the metric argument in the resampling summary returned by summaryFunction. For two-class problems, twoClassSummary returns three metrics: ROC, Sens, and Spec.

A correct setup for sensitivity tuning looks like this:

1library(caret)
2
3ctrl <- trainControl(
4  method = "repeatedcv",
5  number = 5,
6  repeats = 3,
7  classProbs = TRUE,
8  summaryFunction = twoClassSummary
9)
10
11train_df$Class <- factor(train_df$Class, levels = c("event", "other"))
12
13fit <- train(
14  Class ~ .,
15  data = train_df,
16  method = "glm",
17  family = binomial(),
18  trControl = ctrl,
19  metric = "Sens"
20)

If metric = "Sens", caret will select the tuning row with the best resampled sensitivity. It will not silently switch back to ROC unless your configuration still points to metric = "ROC" or the summary function does not produce a Sens column.

Why It Can Still Look Like ROC

There are several reasons the chosen hyperparameters can resemble what ROC optimization would have chosen.

First, better models often improve multiple metrics at the same time. If one parameter set separates the classes more cleanly, both ROC and sensitivity can increase together. In that case the "best for Sens" model may also have the best ROC, and that is expected rather than suspicious.

Second, sensitivity in twoClassSummary is computed for the current class predictions, not for an arbitrary threshold you may have in mind. Many classifiers still convert probabilities to class labels using the default class rule, which often behaves similarly to ROC-improving settings during resampling.

Third, the printed resampling table includes all summary columns. Seeing ROC in the output does not mean ROC was the selection criterion. The deciding factor is the metric argument and the resulting bestTune.

The Positive Class Must Be Defined Correctly

Sensitivity depends on which class is considered the event. In two-class caret workflows, the first factor level is treated as the positive class in many summaries and thresholding helpers.

That means this matters:

train_df$Class <- factor(train_df$Class, levels = c("event", "other"))

If the levels are reversed, you may believe you are optimizing recall for the event class when you are actually optimizing recall for the non-event class. That often looks like "caret is doing the wrong thing" when the labels are simply ordered incorrectly.

A fast diagnostic is to inspect:

levels(train_df$Class)
fit$results
fit$bestTune

If the sensitivity column exists and the class levels are correct, caret is usually doing what you asked.

When You Need Threshold-Specific Sensitivity

A subtler issue is that tuning by sensitivity at the model's default prediction rule is not the same as tuning by sensitivity after you pick a custom probability threshold. If your real workflow is "maximize recall after classifying positive at 0.20 instead of 0.50," then twoClassSummary is not the whole story.

In that situation, use a custom summary function that:

• receives observed classes and predicted probabilities
• applies your chosen threshold
• computes sensitivity from those thresholded predictions

1customSens <- function(data, lev = NULL, model = NULL) {
2  positive_prob <- data[[lev[1]]]
3  pred_class <- ifelse(positive_prob >= 0.20, lev[1], lev[2])
4  pred_class <- factor(pred_class, levels = lev)
5  cm <- confusionMatrix(pred_class, data$obs, positive = lev[1])
6  c(Sens = unname(cm$byClass["Sensitivity"]))
7}

Then pass that function into trainControl(summaryFunction = customSens) and keep metric = "Sens". Now the optimization target matches the actual operating threshold you care about.

Read the Results in the Right Order

When debugging, separate these three questions:

1. What metrics does the summary function compute
2. Which metric does train optimize
3. At what probability threshold are class labels produced

If you answer those in the wrong order, ROC tends to get blamed because it is visible in the output and familiar to practitioners. The real mismatch is usually in the positive class or the thresholding rule.

Common Pitfalls

The most common mistake is setting summaryFunction = twoClassSummary but leaving metric = "ROC". The second most common is forgetting that the first factor level is the positive event, which changes what sensitivity means.

Another frequent problem is assuming that tuning for sensitivity automatically tunes for a non-default probability threshold. It does not unless your summary function applies that threshold explicitly. Finally, do not infer the optimization target from the printed table alone. Check fit$bestTune and the metric setting directly.

Summary

• 'caret optimizes the metric named in train(..., metric = ...), not every metric shown in the results table.'
• With twoClassSummary, the relevant sensitivity metric is Sens.
• Make sure the positive class is the factor level you intend to optimize.
• Sensitivity and ROC can improve together, so similar best models are not proof of wrong optimization.
• Use a custom summary function if you need sensitivity at a specific probability threshold.