Introduction

A decision tree chooses its splitting attribute by testing candidate features and asking a simple question: which split makes the resulting child nodes purer. "Pure" means the rows inside a child node look more alike in terms of the target label or value.

For classification trees, the score is usually based on impurity measures such as Gini impurity or entropy. For regression trees, the tree looks for the split that reduces prediction error, often by minimizing variance or mean squared error.

The Basic Idea Behind a Split

Suppose a node contains examples labeled Yes and No. If the node is mixed, predictions from that node are uncertain. A good split separates the examples so each child node contains more of one class and less of the other.

The tree therefore evaluates many possible questions:

• split on Outlook
• split on Humidity
• split on Wind
• for numeric features, split at thresholds such as temperature <= 24.5

It computes a score for each candidate and chooses the one with the largest impurity reduction.

Gini Impurity and Entropy

Two common classification criteria are Gini impurity and entropy.

For a node containing class probabilities p1, p2, and so on:

• Gini impurity is 1 - sum(pi^2)
• entropy is -sum(pi * log2(pi))

You do not need to memorize the formulas to understand the behavior. Both scores are:

• low when one class dominates the node
• high when classes are evenly mixed

So the tree prefers the split that makes the weighted child impurities as small as possible.

A Small Worked Example

Here is a manual Gini calculation for a toy dataset:

1from collections import Counter
2
3
4def gini(labels):
5    counts = Counter(labels)
6    total = len(labels)
7    return 1.0 - sum((count / total) ** 2 for count in counts.values())
8
9
10parent = ["yes", "yes", "yes", "no", "no", "no"]
11left_child = ["yes", "yes", "no"]
12right_child = ["yes", "no", "no"]
13
14parent_gini = gini(parent)
15weighted_child_gini = (
16    len(left_child) / len(parent) * gini(left_child)
17    + len(right_child) / len(parent) * gini(right_child)
18)
19gain = parent_gini - weighted_child_gini
20
21print("parent gini:", round(parent_gini, 4))
22print("weighted child gini:", round(weighted_child_gini, 4))
23print("gini reduction:", round(gain, 4))

The split with the largest reduction is the one the algorithm prefers.

If the feature is categorical, the candidate split may be based on its values. If the feature is numeric, the algorithm tries thresholds between sorted feature values and picks the best threshold.

Numeric Features Use Threshold Search

For numeric data, the tree does not ask "which attribute" only once. It really asks "which feature and which threshold."

For example:

• 'age <= 30'
• 'age <= 42'
• 'income <= 70000'

Each threshold divides the rows into two groups. The tree scores each candidate threshold and chooses the best one.

This is why training can be expensive on large datasets: a tree is not just choosing from column names, it is exploring many possible cut points inside those columns.

Seeing It in scikit-learn

You can inspect the chosen split after training a small tree:

1from sklearn.tree import DecisionTreeClassifier
2
3X = [
4    [0, 85],
5    [0, 80],
6    [1, 83],
7    [1, 70],
8    [0, 68],
9    [1, 65],
10]
11y = [0, 0, 1, 1, 0, 1]
12
13model = DecisionTreeClassifier(max_depth=1, criterion="gini", random_state=0)
14model.fit(X, y)
15
16print("feature index:", model.tree_.feature[0])
17print("threshold:", model.tree_.threshold[0])

If the output says feature index 1 with a threshold near 74, that means the best first split was on the second feature at approximately that cut point.

Why Different Tree Algorithms Differ

Not all tree algorithms use the same scoring rule:

• CART commonly uses Gini for classification and squared error for regression
• ID3 uses information gain
• C4.5 uses gain ratio to reduce bias toward high-cardinality features

The overall pattern is still the same: evaluate candidate splits, score the child nodes, and pick the best score.

Common Pitfalls

• Thinking the tree picks a split randomly. It is scoring many candidates based on impurity reduction.
• Forgetting that numeric features require threshold search, not just feature selection.
• Assuming Gini and entropy produce completely different trees every time. They are different criteria, but they often select similar splits.
• Ignoring stopping rules such as max_depth or min_samples_split. The best local split is not the whole training story.
• Believing the first split is always the most important feature in a broad causal sense. It is only the best split for that node under the chosen criterion.

Summary

• A decision tree chooses the split that most reduces impurity or prediction error.
• Classification trees commonly use Gini impurity or entropy.
• Numeric features are split by searching candidate thresholds.
• The algorithm scores feature-threshold pairs, not just column names.
• Understanding the split criterion makes tree behavior much easier to interpret.