Introduction

categorical_crossentropy in Keras measures how far a predicted class distribution is from the true one-hot class distribution. It is the standard loss for multi-class classification when each sample belongs to exactly one class. The implementation looks simple mathematically, but in real frameworks it also includes numerical-stability choices and a distinction between probability inputs and logits.

The Core Formula

For one sample with one-hot target y_true and predicted probabilities y_pred, categorical cross-entropy is:

loss = -sum(y_true * log(y_pred))

Because y_true is one-hot in the usual case, this effectively means:

loss = -log(predicted probability of the correct class)

So if the model gives high probability to the correct class, the loss is small. If it gives low probability, the loss becomes large.

What Keras Expects

Keras uses categorical_crossentropy when:

• targets are one-hot encoded
• predictions represent a distribution over classes
• each sample belongs to one class only

Typical model setup:

1import tensorflow as tf
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Dense(16, activation="relu"),
5    tf.keras.layers.Dense(3, activation="softmax"),
6])
7
8model.compile(
9    optimizer="adam",
10    loss="categorical_crossentropy",
11    metrics=["accuracy"],
12)

If your labels are integer class IDs such as 0, 1, 2, then sparse_categorical_crossentropy is usually the better choice instead.

Manual Implementation Example

You can see the basic behavior with TensorFlow directly:

1import tensorflow as tf
2
3y_true = tf.constant([[0.0, 1.0, 0.0]])
4y_pred = tf.constant([[0.1, 0.7, 0.2]])
5
6loss = -tf.reduce_sum(y_true * tf.math.log(y_pred), axis=1)
7print(loss.numpy())

This computes the per-sample loss. For a batch, Keras usually reduces those per-sample values into a mean unless you change the reduction behavior.

Why Numerical Stability Matters

A naive implementation can break when a predicted probability is exactly zero, because log(0) is undefined. Real implementations therefore protect the computation by clipping or by using numerically stable log-softmax logic internally.

That matters especially when the model output is extremely confident or when gradients are computed during training.

In other words, the framework implementation is not just the textbook formula copied literally. It is the textbook formula with practical safeguards.

from_logits=True Versus Softmax Output

One of the most important distinctions is whether the model outputs probabilities or raw logits.

If the final layer already uses softmax, then Keras expects probabilities and from_logits should be false.

If the model outputs raw scores, use the loss object explicitly:

1import tensorflow as tf
2
3loss_fn = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
4
5y_true = tf.constant([[0.0, 1.0, 0.0]])
6logits = tf.constant([[1.2, 2.5, 0.3]])
7
8loss = loss_fn(y_true, logits)
9print(loss.numpy())

This tells Keras to apply the correct stable transformation internally.

Batch Reduction Behavior

Keras computes the loss per example and then reduces it across the batch. In everyday training, that reduction is usually the mean. Understanding that detail helps when you compare framework loss values against a manual calculation and get slightly different-looking outputs because one result is per-sample and the other is already batch-reduced.

Common Pitfalls

• Using categorical_crossentropy with integer labels instead of one-hot labels.
• Applying a softmax layer and also setting from_logits=True.
• Reimplementing the loss naively and running into log(0) instability.
• Assuming the returned loss is always per-sample when Keras may already have reduced it across the batch.
• Using categorical cross-entropy for a problem that is actually multi-label rather than single-label multi-class.

Summary

• 'categorical_crossentropy is the negative log-likelihood of the correct class under a predicted class distribution.'
• In Keras it is intended for one-hot multi-class targets.
• The framework adds numerical-stability protection beyond the raw textbook formula.
• 'from_logits=True is for raw scores, not softmax probabilities.'
• Always match the loss choice to the label representation and model output type.