Introduction

In R, the randomForest package can return either class labels or class probabilities for classification tasks. Probability output is essential for threshold tuning, ranking, and calibration workflows. The key is to train a classification forest with a factor target and call predict with type = "prob".

Core Sections

Train a classification model

Your response variable must be a factor for classification mode.

1library(randomForest)
2
3set.seed(42)
4data(iris)
5
6model <- randomForest(
7  Species ~ ., 
8  data = iris,
9  ntree = 300,
10  mtry = 2,
11  importance = TRUE
12)
13
14print(model)

If target is numeric, randomForest switches to regression and probability output will not apply.

Get probability predictions

Use predict with type = "prob".

prob <- predict(model, newdata = iris[1:5, ], type = "prob")
print(prob)

The result is a matrix with one column per class. Each row sums to one.

Convert probabilities to class with custom threshold

Default classification chooses highest probability, but custom thresholds are useful for imbalanced data.

1prob <- predict(model, newdata = iris, type = "prob")
2
3pred_custom <- ifelse(prob[, "virginica"] >= 0.60, "virginica", "other")
4table(pred_custom)

Threshold tuning lets you trade precision and recall based on business costs.

Evaluate probability quality

Accuracy alone is not enough for probability models. Add metrics like log loss, ROC AUC, and calibration diagnostics.

1library(pROC)
2
3is_virginica <- as.integer(iris$Species == "virginica")
4auc_val <- auc(is_virginica, prob[, "virginica"])
5print(auc_val)

Well-ranked probabilities are useful even when class labels at default threshold are imperfect.

Understand class ordering and column names

Probability columns follow factor level order. Always inspect levels(y) and column names before downstream mapping.

levels(iris$Species)
colnames(prob)

Incorrect column assumptions are a common source of silent errors in reporting pipelines.

Production usage guidance

Persist both model object and class level metadata together. During scoring, validate that new data uses compatible feature names and types. For highly imbalanced problems, consider class weights or sampling strategies during training, then evaluate calibrated probabilities on a holdout dataset.

In regulated or high-impact settings, log predicted probabilities and final decision thresholds used at inference time. This improves auditability and post-incident analysis.

Train and score with holdout data

Probability output is most useful when evaluated on unseen data. Split data into train and test, then compute probability metrics on the test set.

1set.seed(7)
2idx <- sample(seq_len(nrow(iris)), size = 0.8 * nrow(iris))
3train <- iris[idx, ]
4test  <- iris[-idx, ]
5
6model <- randomForest(Species ~ ., data = train, ntree = 400)
7prob_test <- predict(model, newdata = test, type = "prob")
8head(prob_test)

Then evaluate class predictions with a chosen threshold or argmax strategy.

pred_label <- colnames(prob_test)[max.col(prob_test, ties.method = "first")]
table(pred_label, test$Species)

For binary problems, store the probability for the positive class and compare against business-defined thresholds. This enables cost-sensitive decisioning and supports downstream risk ranking. Probability-first workflows are usually more flexible than fixed-label outputs.

Store threshold values in configuration instead of hardcoding them in scoring scripts. This allows safe tuning without retraining and keeps decision policy changes auditable over time.

Document class label order in model artifacts so probability columns are interpreted correctly by downstream services and dashboards.

Include this check in CI scoring tests.

Common Pitfalls

• Training with numeric target and expecting classification probabilities.
• Forgetting type = "prob" and receiving class labels instead.
• Misreading probability columns due to incorrect factor-level assumptions.
• Using default threshold blindly in imbalanced classification tasks.
• Evaluating only accuracy and ignoring probability calibration quality.

Summary

• Use factor targets and predict(..., type = "prob") for class probabilities.
• Validate column-to-class mapping through factor levels and column names.
• Tune thresholds according to precision and recall tradeoffs.
• Evaluate ranking and calibration, not only class accuracy.
• Preserve model metadata to keep scoring behavior consistent in production.