Introduction

If duplicate samples can appear in both train and test sets, evaluation becomes overly optimistic because the model is effectively seeing the same example twice. The correct fix is not a normal row-wise split. It is a group-based split where all duplicates are assigned to the same side.

Why Ordinary train_test_split Fails

A row-wise split treats every row as independent. That breaks down when multiple rows represent the same underlying sample.

Example problem:

1import pandas as pd
2
3df = pd.DataFrame({
4    "sample_id": [1, 1, 2, 2, 3, 4],
5    "x": [10, 10, 20, 20, 30, 40],
6    "y": [0, 0, 1, 1, 0, 1],
7})
8
9print(df)

If you split this row by row, one copy of sample 1 might go to training and another to testing. That leaks information.

Split by Group Instead of by Row

The right abstraction is a group identifier that marks all duplicates belonging to the same underlying sample. If you already have a sample ID, use that. If not, build one from the duplicated feature columns.

With scikit-learn, GroupShuffleSplit is often the simplest solution.

1from sklearn.model_selection import GroupShuffleSplit
2
3X = df[["x"]]
4y = df["y"]
5groups = df["sample_id"]
6
7splitter = GroupShuffleSplit(test_size=0.33, n_splits=1, random_state=42)
8train_idx, test_idx = next(splitter.split(X, y, groups=groups))
9
10train_df = df.iloc[train_idx]
11test_df = df.iloc[test_idx]
12
13print(train_df)
14print(test_df)

This guarantees that all rows with the same sample_id stay together.

Build a Group Key When No ID Exists

Sometimes the dataset has duplicates but no explicit identifier. In that case, construct a group key from the columns that define duplication.

1import pandas as pd
2
3raw = pd.DataFrame({
4    "feature_a": [1, 1, 2, 2, 3],
5    "feature_b": [5, 5, 6, 6, 7],
6    "label": [0, 0, 1, 1, 0],
7})
8
9raw["group_key"] = raw[["feature_a", "feature_b", "label"]].astype(str).agg("|".join, axis=1)
10print(raw)

If duplicates are defined only by features and not by label, leave the label out of the key. The point is to capture the real overlap rule in one grouping column.

Then split on that key the same way.

Use Grouped Cross-Validation Too

If you are doing model selection rather than a single holdout split, use grouped cross-validation instead of ordinary K-fold.

1from sklearn.model_selection import GroupKFold
2
3cv = GroupKFold(n_splits=3)
4
5for fold, (train_idx, test_idx) in enumerate(cv.split(X, y, groups=groups), start=1):
6    train_groups = set(groups.iloc[train_idx])
7    test_groups = set(groups.iloc[test_idx])
8    print(fold, train_groups.intersection(test_groups))

The intersection should be empty for each fold. That is the guarantee you actually want.

Verify the Split Explicitly

Do not assume the grouping logic is correct. Check it.

1train_groups = set(train_df["sample_id"])
2test_groups = set(test_df["sample_id"])
3
4overlap = train_groups.intersection(test_groups)
5print(overlap)

If the overlap is non-empty, the split is still leaking duplicates.

This verification step is especially important when the grouping key was built manually from several columns.

Stratification and Grouping Can Conflict

A common follow-up requirement is preserving class balance while also respecting duplicate groups. That is harder than plain group splitting because perfect class proportions may not be possible once groups are indivisible.

If both matter, consider:

• 'StratifiedGroupKFold if your sklearn version provides it'
• custom group-aware splitting logic
• accepting approximate rather than perfect label balance

The higher priority is usually no leakage. A slightly imperfect class ratio is often better than a perfectly balanced but contaminated test set.

Common Pitfalls

• Using ordinary train_test_split on duplicated data and assuming randomization is enough.
• Defining duplicate groups incorrectly, so near-identical rows still leak across the split.
• Forgetting to use grouped cross-validation after fixing the holdout split.
• Prioritizing exact class balance over eliminating overlap leakage.
• Skipping explicit overlap checks after the split.

Summary

• Duplicate samples should be split by group, not by individual row.
• 'GroupShuffleSplit is a strong default for train-test splitting without overlap.'
• Build a group key manually if no sample ID exists.
• Use grouped cross-validation for model selection as well.
• Always verify that no group appears in both train and test sets.