Introduction

In machine learning projects, whether to split train and test sets before exploratory data analysis is a critical process decision. The safest default is to create holdout sets early and protect them from model-driven decisions. You can still do useful exploration, but the test set should remain a final evaluation artifact, not a design input.

Core Sections

Why split timing matters

If test data influences feature engineering, model selection, or preprocessing decisions, evaluation becomes optimistic. This is data leakage, even when no labels are explicitly leaked in code.

Leakage effects:

• inflated offline metrics
• disappointing production performance
• fragile models that do not generalize

Splitting early reduces this risk significantly.

Recommended workflow in practice

A robust flow:

1. Define target and leakage-sensitive features.
2. Create immutable test split first.
3. Perform EDA mostly on training data.
4. Build preprocessing and models using only training folds.
5. Use validation and cross-validation for iterative tuning.
6. Evaluate once on untouched test set.

This keeps the test set as an honest simulation of unseen future data.

Code example with early split

1from sklearn.model_selection import train_test_split
2import pandas as pd
3
4# df has features and target column 'label'
5X = df.drop(columns=["label"])
6y = df["label"]
7
8X_train, X_test, y_train, y_test = train_test_split(
9    X, y, test_size=0.2, random_state=42, stratify=y
10)
11
12# EDA for modeling decisions should focus on X_train and y_train
13print(X_train.shape, X_test.shape)

If class imbalance exists, stratified split is important to keep class proportions stable.

What EDA can include before split

Some teams still run high-level sanity checks on full data before splitting, which can be acceptable if it does not influence model tuning.

Low-risk pre-split checks:

• schema and datatype validation
• missing-value rates
• duplicate row counts
• basic target distribution sanity

High-risk if done pre-split and used for decisions:

• feature selection based on full correlation with target
• outlier removal thresholds tuned from full set
• encoding or scaling statistics fitted on full set

Pipelines prevent leakage

Use scikit-learn pipelines so fit steps run only on training folds.

1from sklearn.pipeline import Pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.linear_model import LogisticRegression
4from sklearn.model_selection import cross_val_score
5
6pipeline = Pipeline([
7    ("scaler", StandardScaler()),
8    ("model", LogisticRegression(max_iter=1000)),
9])
10
11scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="roc_auc")
12print(scores.mean())

This ensures scaling parameters come from training folds only.

Time-series and grouped data caveats

Random split is not always valid.

For time-series:

• split by time order
• avoid future leakage into past

For grouped entities such as users or patients:

• split by group ids to avoid identity leakage

Split strategy must reflect prediction-time reality.

Role of validation set versus test set

Use validation for model iteration. Use test set once at the end for final estimate.

If you repeatedly check test metrics during development, the test set effectively becomes validation data and loses unbiased status. In that case, create a new final holdout if possible.

Documentation and governance

Record in project docs:

• exact split strategy
• random seed and stratification policy
• time cutoff or grouping rules
• leakage assumptions and excluded features

This improves reproducibility and auditability in collaborative ML teams.

Common Pitfalls

• Running feature selection on full dataset before train-test split.
• Fitting scalers or imputers on full data and leaking statistics.
• Repeatedly tuning models using test-set feedback.
• Using random split for time-series prediction tasks.
• Ignoring entity-level leakage in grouped datasets.

Summary

• Split early and protect the test set from iterative model decisions.
• Do detailed EDA and preprocessing fitting on training data only.
• Use pipelines and cross-validation to reduce leakage risk.
• Choose split strategy that matches deployment scenario, especially for time and grouped data.
• Treat final test evaluation as one-time confirmation of generalization.