Introduction

TreeBagger is MATLAB's ensemble learner for bagged decision trees, and it is often used to build random forests for classification and regression. The class exposes many parameters, but a few of them drive most of the model's behavior: tree count, tree depth, feature sampling, and out-of-bag evaluation.

Start With the Parameters That Matter Most

A random forest combines many decision trees trained on bootstrap samples of the data. In MATLAB, that starts with the number of trees:

• 'NumTrees controls ensemble size.'
• 'MinLeafSize limits how small terminal leaves can become.'
• 'NumPredictorsToSample controls feature randomness at each split.'
• 'OOBPrediction enables out-of-bag error estimates.'

These parameters trade accuracy, variance, speed, and interpretability against one another.

1load fisheriris
2X = meas;
3Y = species;
4
5rng(7)
6Mdl = TreeBagger(150, X, Y, ...
7    'Method', 'classification', ...
8    'MinLeafSize', 5, ...
9    'NumPredictorsToSample', 2, ...
10    'OOBPrediction', 'on');
11
12oobErr = oobError(Mdl);
13plot(oobErr)
14xlabel('Number of grown trees')
15ylabel('Out-of-bag classification error')

This example is a good baseline because it shows whether adding more trees is still improving performance.

Understanding Core Tuning Knobs

NumTrees

More trees usually stabilize predictions and reduce variance, but after a point the gains flatten out. If the out-of-bag curve plateaus after 120 trees, growing 1000 trees may add little value besides slower training and larger models.

MinLeafSize

Small leaves let each tree fit fine-grained patterns. That can be useful when signal is subtle, but it also makes individual trees noisy. Increasing MinLeafSize usually produces shallower, smoother trees that generalize better when the dataset is small or noisy.

NumPredictorsToSample

This parameter is central to the "random" part of random forests. At each split, only a subset of predictors is considered. Smaller values increase diversity between trees; larger values let each tree search more aggressively for the best split.

For classification, a common starting point is roughly the square root of the predictor count. For regression, a common starting point is around one third of the predictors. Those are heuristics, not laws.

Parameters That Help You Diagnose the Model

OOBPrediction is one of the most practical settings in TreeBagger. Since each tree is trained on a bootstrap sample, some observations are left out for that tree. Those left-out rows can be used as an internal validation set.

For feature relevance, enable permutation-based importance:

1rng(7)
2Mdl = TreeBagger(200, X, Y, ...
3    'Method', 'classification', ...
4    'OOBPrediction', 'on', ...
5    'OOBPredictorImportance', 'on');
6
7bar(Mdl.OOBPermutedPredictorDeltaError)
8set(gca, 'XTickLabel', {'SepalL','SepalW','PetalL','PetalW'})
9ylabel('Importance score')

A larger importance score means that permuting that predictor hurt out-of-bag performance more.

Controlling Tree Growth and Sampling

Two other parameters often matter in real projects.

MaxNumSplits limits the number of branch decisions in each tree. This is another way to constrain complexity.

InBagFraction changes how much of the dataset is sampled for each tree. Standard bagging uses bootstrap sampling, but lowering the in-bag fraction can sometimes improve speed or diversity.

For imbalanced classification, Prior or observation weights can prevent the forest from over-optimizing for the majority class.

1Mdl = TreeBagger(300, X, Y, ...
2    'Method', 'classification', ...
3    'Prior', 'uniform', ...
4    'MinLeafSize', 3, ...
5    'MaxNumSplits', 20);

The right choice depends on the cost of false positives versus false negatives, not just raw accuracy.

Regression Uses the Same Ideas

For regression, the interface is almost identical, but the forest predicts averages instead of class votes.

1rng(1)
2X = randn(200, 5);
3y = 3 * X(:,1) - 2 * X(:,3) + 0.3 * randn(200, 1);
4
5Mdl = TreeBagger(100, X, y, ...
6    'Method', 'regression', ...
7    'MinLeafSize', 8, ...
8    'OOBPrediction', 'on');
9
10pred = predict(Mdl, X(1:5,:));
11disp(pred)

The same tuning logic applies: more trees reduce variance, leaf size controls local fit, and out-of-bag error helps estimate generalization.

Common Pitfalls

A common mistake is treating NumTrees as the only important parameter. If trees are too deep or too shallow, adding more of them will not fix the underlying bias or variance problem.

Another issue is reading too much into feature importance scores. Correlated predictors can split importance across multiple columns, so a low score does not automatically mean a feature is useless.

Developers also forget that out-of-bag estimates are diagnostics, not magic. For time-series or grouped data, random bootstrap sampling may violate the structure of the problem, so a custom validation split can still be necessary.

Summary

• 'NumTrees, MinLeafSize, and NumPredictorsToSample control most forest behavior.'
• 'OOBPrediction gives a fast built-in estimate of generalization error.'
• Use OOBPredictorImportance to inspect feature relevance, but interpret it carefully.
• Constraining tree growth can improve robustness and training speed.
• Tune for the data distribution and error costs, not for defaults alone.