Introduction

Overfitting happens when a network learns the training set too specifically and fails to generalize to new examples. A simple feed-forward network can still overfit if the dataset is small, noisy, or if the model is trained too long with too much capacity.

Recognize the Pattern First

The classic symptom is a widening gap between training and validation metrics. Training loss keeps falling, but validation loss flattens out or starts rising. That means the model is memorizing details that do not help on unseen data.

A good workflow starts with a validation set and a training curve. Without those, you are guessing.

Start with a Smaller Network

Many beginners respond to weak results by adding more layers and more units. That often makes overfitting worse. A smaller model is easier to train and harder to memorize with.

Here is a compact Keras example for binary classification:

1import tensorflow as tf
2from sklearn.datasets import make_classification
3from sklearn.model_selection import train_test_split
4from sklearn.preprocessing import StandardScaler
5
6X, y = make_classification(
7    n_samples=2000,
8    n_features=20,
9    n_informative=10,
10    n_redundant=5,
11    random_state=42,
12)
13
14X_train, X_test, y_train, y_test = train_test_split(
15    X, y, test_size=0.2, random_state=42
16)
17
18scaler = StandardScaler()
19X_train = scaler.fit_transform(X_train)
20X_test = scaler.transform(X_test)
21
22model = tf.keras.Sequential([
23    tf.keras.layers.Dense(
24        32,
25        activation="relu",
26        kernel_regularizer=tf.keras.regularizers.l2(1e-4),
27        input_shape=(X_train.shape[1],),
28    ),
29    tf.keras.layers.Dropout(0.2),
30    tf.keras.layers.Dense(1, activation="sigmoid"),
31])
32
33model.compile(
34    optimizer="adam",
35    loss="binary_crossentropy",
36    metrics=["accuracy"],
37)

This model is intentionally modest. One hidden layer is often enough for tabular data baselines.

Use Validation Data and Early Stopping

The most practical anti-overfitting tool is early stopping. Train for many epochs, but stop once validation loss stops improving.

1callbacks = [
2    tf.keras.callbacks.EarlyStopping(
3        monitor="val_loss",
4        patience=10,
5        restore_best_weights=True,
6    )
7]
8
9history = model.fit(
10    X_train,
11    y_train,
12    validation_split=0.2,
13    epochs=200,
14    batch_size=32,
15    callbacks=callbacks,
16    verbose=0,
17)
18
19test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
20print(f"Test accuracy: {test_acc:.3f}")

This prevents the model from drifting into memorization late in training. It is cheap, easy to add, and should be part of almost every baseline experiment.

Regularize Instead of Guessing

Two regularization methods work especially well for simple feed-forward networks:

• 'Dropout, which randomly disables a fraction of activations during training'
• 'L2 weight decay, which discourages large weights'

Both are already present in the example above. The goal is not to make the model weak. The goal is to make it less brittle.

Do not overdo either one. Very aggressive dropout or regularization can cause underfitting, where both training and validation performance stay poor.

Data Quality Often Matters More Than Architecture

If your model overfits badly, the dataset may be the real bottleneck. Typical causes include:

• too few samples
• label noise
• duplicated records
• leakage between train and validation sets

For tabular data, better splitting and cleaning usually help more than stacking extra dense layers. If your dataset contains near-duplicates across training and validation, the validation score can look good during experimentation and then collapse in production.

Standardization also matters. Dense networks are easier to train when numeric features are scaled consistently, which is why the example uses StandardScaler.

A Good Baseline Strategy

A disciplined process usually works better than a clever network:

1. start with one or two dense layers
2. keep the hidden width moderate
3. scale the inputs
4. add a validation split
5. use early stopping
6. only then tune dropout, weight decay, and layer size

That order helps you tell whether the improvement comes from better generalization or just more capacity.

Common Pitfalls

• Evaluating only on the training set and assuming a high accuracy means the model is good.
• Making the network deeper before checking whether the data split is sound.
• Applying scaling before the train-test split, which leaks information from the test set.
• Using too much dropout and then mistaking underfitting for overfitting.
• Training for a fixed large number of epochs without early stopping or validation monitoring.

Summary

• Overfitting shows up when training performance improves while validation performance worsens.
• A smaller dense network is often a stronger baseline than a larger one.
• Early stopping is one of the simplest and most effective defenses.
• Use regularization thoughtfully, not automatically.
• Clean splits, scaled inputs, and good data quality usually matter as much as model architecture.