Introduction

When a simple MLP fails to converge in TensorFlow, the problem is usually not that the model is too small. More often, the issue is a mismatch between data, labels, output layer, and loss function, or an optimization setup that makes learning unstable.

Start with a small correct baseline

Before tuning layers or activations, make sure you can train a minimal model with a sensible configuration. For binary classification, a reliable baseline looks like this:

1import numpy as np
2import tensorflow as tf
3
4x = np.random.rand(1000, 10).astype("float32")
5y = np.random.randint(0, 2, size=(1000, 1)).astype("float32")
6
7model = tf.keras.Sequential([
8    tf.keras.layers.Input(shape=(10,)),
9    tf.keras.layers.Dense(32, activation="relu"),
10    tf.keras.layers.Dense(16, activation="relu"),
11    tf.keras.layers.Dense(1, activation="sigmoid"),
12])
13
14model.compile(
15    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
16    loss="binary_crossentropy",
17    metrics=["accuracy"],
18)
19
20history = model.fit(x, y, epochs=10, batch_size=32, validation_split=0.2)

This does not guarantee good accuracy on real data, but it does give you a reference point. If your training code is very different from this baseline, simplify it until you know the training pipeline is logically correct.

Make the output layer match the loss

One of the most common convergence failures is using an inconsistent output-loss pair.

For binary classification, typical choices are:

• one output unit with sigmoid and binary_crossentropy
• one linear output with BinaryCrossentropy(from_logits=True)

For multiclass classification, typical choices are:

• 'softmax with categorical_crossentropy for one-hot labels'
• 'softmax with sparse_categorical_crossentropy for integer labels'

If these pieces do not match, training can run without crashing while still learning almost nothing.

Scale the input features

MLPs are very sensitive to feature scale. If one input column is between 0 and 1 and another is in the millions, gradient updates become hard to interpret and optimization slows down or becomes unstable.

A simple normalization pass often helps immediately:

1import numpy as np
2
3x_mean = x.mean(axis=0, keepdims=True)
4x_std = x.std(axis=0, keepdims=True) + 1e-8
5x_scaled = (x - x_mean) / x_std

Train the same model on x_scaled and compare the loss curve. For tabular data, feature scaling is often more important than adding extra hidden layers.

Check the learning rate before changing the architecture

If the learning rate is too high, the loss can jump around or diverge. If it is too low, the loss may look flat and training seems stuck. Do a small controlled experiment before changing the network depth:

1model.compile(
2    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
3    loss="binary_crossentropy",
4    metrics=["accuracy"],
5)

If training becomes stable after lowering the learning rate, the architecture was probably not the real issue.

Verify labels and shapes explicitly

TensorFlow can sometimes broadcast arrays in ways that let code run even though the setup is logically wrong. Inspect your training arrays directly:

print(x.shape)
print(y.shape)
print(y.dtype)

For a single sigmoid output, labels shaped like n x 1 with float32 values are a clean default. If your labels are integers for a multiclass task, make sure the loss function expects integer class indices.

Overfit a tiny subset on purpose

One of the best debugging tests is to train on a tiny subset and see whether the model can overfit it:

1small_x = x[:64]
2small_y = y[:64]
3
4model.fit(small_x, small_y, epochs=100, batch_size=16, verbose=0)

If the network cannot fit a tiny subset, the training setup is usually broken. That points you back toward labels, loss configuration, data scaling, or optimizer settings rather than model capacity.

Common Pitfalls

• Changing model depth before checking the loss function, label encoding, and feature scaling.
• Using an output layer that does not match the selected loss.
• Training on unscaled tabular features and assuming the optimizer will handle every magnitude difference.
• Looking only at accuracy instead of following training and validation loss.
• Ignoring the tiny-subset overfit test, which often exposes configuration bugs quickly.

Summary

• Most MLP convergence failures in TensorFlow come from setup errors rather than insufficient complexity.
• Match the output layer, label encoding, and loss function carefully.
• Normalize features before spending time on architecture changes.
• Check the learning rate and use a tiny-subset overfit test to isolate broken training setups.