Introduction

Splitting data into training, validation, and test sets is one of the first decisions that affects whether a machine learning evaluation is trustworthy. The goal is simple: train on one subset, tune on another, and evaluate once on a truly unseen subset. The implementation is easy, but the data leakage rules and sampling choices matter more than the syntax.

What Each Split Is For

The three-way split exists because each subset serves a different purpose:

• training set: used to fit model parameters
• validation set: used to tune hyperparameters and compare model variants
• test set: used only for final evaluation

The test set should stay untouched during model tuning. If you repeatedly inspect test results while changing the model, the test set stops being an honest estimate of generalization.

A Simple scikit-learn Pattern

The most common implementation is to split twice with train_test_split.

Example with a 70/15/15 split:

1from sklearn.model_selection import train_test_split
2
3X = list(range(100))
4y = [value % 2 for value in X]
5
6X_train, X_temp, y_train, y_temp = train_test_split(
7    X, y, test_size=0.30, random_state=42, stratify=y
8)
9
10X_val, X_test, y_val, y_test = train_test_split(
11    X_temp, y_temp, test_size=0.50, random_state=42, stratify=y_temp
12)
13
14print(len(X_train), len(X_val), len(X_test))

The first split creates:

• '70% training'
• '30% temporary holdout'

The second split divides that temporary holdout equally into validation and test, giving 15% each.

Why stratify Is Often Important

For classification problems, class balance matters. If the target is imbalanced, a purely random split can accidentally create uneven label distributions across the subsets.

That is why stratify=y is often a good default for classification:

train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

This helps preserve class proportions in each subset.

Without stratification, the validation or test set may become misleading, especially when minority classes are rare.

Reproducibility with random_state

Always set a fixed random seed when you want reproducible experiments:

random_state=42

This does not make the split universally correct, but it ensures that:

• teammates can reproduce your split
• repeated experiments stay comparable
• debugging is easier

If every run creates a different split, performance changes become harder to interpret.

Choose Ratios Based on Data Size

There is no universal perfect ratio. Common choices include:

• '80/10/10'
• '70/15/15'
• '60/20/20'

The best choice depends on:

• total dataset size
• class balance
• how expensive validation is
• how much data the model needs to learn effectively

For very large datasets, a smaller validation and test fraction can still be enough. For small datasets, you may need cross-validation instead of relying on one fixed split.

Time Series and Grouped Data Need Different Rules

Random splitting is not always valid.

For time series, preserve temporal order:

• train on earlier data
• validate on later data
• test on the latest data

For grouped data such as user sessions, patients, or devices, make sure related records do not leak across splits. If the same user appears in both training and test, your evaluation may look better than real deployment performance.

So the real rule is not "always random split." The real rule is "split in a way that matches the deployment scenario."

Avoid Data Leakage in Preprocessing

Even if the row split is correct, leakage can still happen during preprocessing.

Bad pattern:

1. fit a scaler on the full dataset
2. split later

Better pattern:

1. split first
2. fit preprocessing only on the training set
3. apply the learned transform to validation and test

This rule applies to:

• scaling
• imputation
• feature selection
• target encoding

If a preprocessing step learns from the whole dataset, your validation and test results are contaminated.

A Practical Workflow

In many projects, a good default workflow is:

1. split the raw data into train, validation, and test
2. fit preprocessing on train only
3. tune models with validation
4. evaluate once on test

That structure is more important than the exact percentage points of the split.

If model comparison is unstable, you may need cross-validation or repeated splits, but the core separation of responsibilities still applies.

Common Pitfalls

The biggest mistake is using the test set during model tuning. Once that happens, test performance stops being an honest final estimate.

Another issue is forgetting stratification for imbalanced classification tasks, which can produce misleading validation and test subsets.

Developers also often fit preprocessing on the full dataset before splitting, which leaks information from validation and test into training.

Finally, random splitting is wrong for some data types such as time series or grouped records. The split strategy should reflect how the model will actually be used.

Summary

• Split data so training, validation, and test each serve a distinct purpose.
• A common implementation is two calls to train_test_split.
• Use stratification for classification when class balance matters.
• Fit preprocessing on the training set only to avoid leakage.
• For time series or grouped data, do not assume random splitting is valid.