Introduction

When you combine class imbalance with grouped data, the order of operations matters. In Leave-One-Group-Out cross-validation, oversampling must happen inside each training fold only, otherwise you leak information from the held-out group into the model-selection process.

The Correct Rule

For each LOGO split:

1. keep one group completely untouched as the test fold
2. take all remaining groups as training data
3. run oversampling only on that training portion
4. fit the model
5. evaluate on the untouched group

That is the entire principle. If you oversample before the split, synthetic samples will be influenced by data that should have remained held out.

A Safe Implementation with imblearn

Using imblearn.pipeline.Pipeline is the cleanest approach because the sampler is applied during fit and not during predict. The pipeline below is fully runnable if you have scikit-learn and imbalanced-learn installed.

1import numpy as np
2from imblearn.over_sampling import SMOTE
3from imblearn.pipeline import Pipeline
4from sklearn.datasets import make_classification
5from sklearn.linear_model import LogisticRegression
6from sklearn.metrics import balanced_accuracy_score
7from sklearn.model_selection import LeaveOneGroupOut
8
9X, y = make_classification(
10    n_samples=240,
11    n_features=8,
12    n_informative=4,
13    n_redundant=0,
14    n_classes=2,
15    weights=[0.88, 0.12],
16    random_state=42,
17)
18
19groups = np.repeat(np.arange(12), 20)
20logo = LeaveOneGroupOut()
21
22pipeline = Pipeline(
23    [
24        ("smote", SMOTE(random_state=42, k_neighbors=3)),
25        ("model", LogisticRegression(max_iter=1000)),
26    ]
27)
28
29scores = []
30
31for train_idx, test_idx in logo.split(X, y, groups):
32    pipeline.fit(X[train_idx], y[train_idx])
33    preds = pipeline.predict(X[test_idx])
34    score = balanced_accuracy_score(y[test_idx], preds)
35    scores.append(score)
36
37print("fold scores:", [round(s, 3) for s in scores])
38print("mean score:", round(float(np.mean(scores)), 3))

The important detail is that SMOTE sees only X[train_idx] and y[train_idx] inside each fold.

Why Pre-Oversampling Is Wrong

Suppose group 7 is the held-out group for one fold. If you run SMOTE on the full dataset first, the synthetic training samples may be interpolated using points from group 7. Even though you later separate the indices, the training data has already been contaminated by the test group.

That makes the score optimistic. The whole reason to use LOGO is to test generalization across groups, so leaking group information destroys the point of the validation strategy.

Watch the Minority Count in Each Fold

SMOTE has another practical constraint: it needs enough minority samples in the current training fold. If one split contains very few minority examples, k_neighbors=5 may be impossible.

A defensive pattern is to adapt the sampler per fold:

1import numpy as np
2from collections import Counter
3from imblearn.over_sampling import RandomOverSampler, SMOTE
4
5
6def make_sampler(y_train):
7    minority_count = min(Counter(y_train).values())
8    if minority_count < 2:
9        return RandomOverSampler(random_state=42)
10    return SMOTE(random_state=42, k_neighbors=min(3, minority_count - 1))

That way the fold does not crash just because the minority class is tiny in one group split.

Pipeline Design Matters

If you also scale features, select features, or tune hyperparameters, keep those steps inside the fold as well. A common pattern is sampler plus model in an imblearn pipeline, then manual LOGO iteration or a group-aware search procedure.

The rule is consistent: anything that learns from data must be fit only on the training groups for that fold.

Common Pitfalls

• Oversampling the full dataset before LOGO splitting. That is leakage.
• Using a plain sklearn.pipeline.Pipeline with a sampler. Samplers need imblearn.pipeline.Pipeline.
• Forgetting that some folds may have too few minority examples for the default SMOTE neighbor count.
• Reporting ordinary accuracy on a severely imbalanced problem and calling the evaluation complete.
• Ignoring the meaning of groups. If groups represent subjects, sessions, or devices, leakage across them is exactly what LOGO is designed to prevent.

Summary

• Apply oversampling after the LOGO split, never before it.
• The sampler must see only the training groups in each fold.
• 'imblearn.pipeline.Pipeline is the cleanest implementation pattern.'
• Check minority counts per fold because SMOTE may need a smaller k_neighbors or a fallback sampler.
• Group-aware validation is only useful if every preprocessing step respects the group boundary.