Introduction

When a simple Keras model is not learning, the cause is usually basic rather than exotic. Most failures come from mismatched output layers and loss functions, unscaled inputs, label problems, or a training setup that has never been validated on a tiny overfit test.

Start With a Known-Good Baseline

Before changing the architecture, make sure the overall training loop is sane. A minimal binary classification model should look something like this:

1import numpy as np
2import tensorflow as tf
3
4x = np.random.rand(500, 4).astype("float32")
5y = (x.sum(axis=1) > 2).astype("float32")
6
7model = tf.keras.Sequential(
8    [
9        tf.keras.layers.Input(shape=(4,)),
10        tf.keras.layers.Dense(8, activation="relu"),
11        tf.keras.layers.Dense(1, activation="sigmoid"),
12    ]
13)
14
15model.compile(
16    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
17    loss="binary_crossentropy",
18    metrics=["accuracy"],
19)
20
21history = model.fit(x, y, epochs=10, batch_size=32, verbose=0)
22print(history.history["loss"][-1])

If this kind of setup works on synthetic data but your real problem does not, the issue is likely in the data or problem formulation rather than in Keras itself.

Check Output Layer and Loss Compatibility

A very common mistake is mixing the wrong output activation with the wrong loss.

Examples:

• Binary classification: one output unit plus sigmoid plus binary_crossentropy.
• Multi-class single-label classification: softmax plus sparse_categorical_crossentropy or categorical_crossentropy.
• Regression: usually no output activation or a task-specific one, plus a regression loss such as mse.

A mismatch here can make learning look broken even though the optimizer is running exactly as asked.

Scale Inputs Before Training

Simple dense networks are sensitive to input scale. If one feature ranges from 0 to 1 and another ranges from 0 to 100000, optimization may become unstable or slow.

A quick normalization step is often enough:

mean = x.mean(axis=0, keepdims=True)
std = x.std(axis=0, keepdims=True) + 1e-7
x_scaled = (x - mean) / std

If the network only starts learning after scaling, that is not a surprise. It is normal.

Verify the Labels, Not Just the Features

If labels are wrong, inconsistent, or misaligned with the feature rows, the network cannot learn the intended mapping. Check:

• Are labels shuffled independently from features?
• Are class IDs in the expected range?
• Are there unexpected NaN values?
• Does the target actually depend on the provided features?

A model that stays near chance accuracy is often reflecting a data-label problem instead of an optimizer problem.

Run the Tiny Overfit Test

One of the best diagnostics is to train on a very small subset, such as 20 samples, and see whether the network can nearly memorize it.

1small_x = x[:20]
2small_y = y[:20]
3
4model.fit(small_x, small_y, epochs=200, verbose=0)
5print(model.evaluate(small_x, small_y, verbose=0))

If the model cannot overfit a tiny clean subset, something fundamental is wrong:

• Loss and output mismatch.
• Broken labels.
• Learning rate issue.
• Data preprocessing bug.

This test is often more informative than adding layers or training longer.

Learning Rate Still Matters

A learning rate that is too high can make the loss bounce or diverge. Too low can make the network appear frozen.

The default Adam settings are a good starting point, but if training is flat, try a modest sweep instead of guessing forever:

• '1e-2'
• '1e-3'
• '1e-4'

Do not change ten things at once. Change one variable and observe the loss curve.

Common Pitfalls

• Using an output layer and loss function that do not match the task.
• Feeding unscaled numeric inputs into a dense network.
• Training on misaligned or low-quality labels.
• Assuming the model is the problem before running a tiny overfit test.
• Changing architecture repeatedly without checking the basics first.

Summary

• Most simple Keras learning failures come from setup errors, not from the framework.
• Check output activation and loss compatibility first.
• Scale inputs and verify label integrity.
• Use a tiny overfit test to prove the training loop can learn at all.
• Tune learning rate only after the data and task formulation make sense.