Introduction

Keras's CategoricalCrossentropy is one of the most commonly used loss functions for multi-class classification tasks. It measures how different the predicted probability distribution is from the true label distribution, guiding the model's weights toward more accurate predictions during training. This article explains exactly what the function computes, walks through a concrete example, and covers practical considerations for using it correctly.

What Is Cross-Entropy?

Cross-entropy comes from information theory. It measures the average number of bits needed to encode events from one probability distribution using a code optimized for a different distribution. In machine learning, we use it to measure how well the predicted distribution $\hat{y}$ matches the true distribution $y$.

The lower the cross-entropy, the closer the predicted distribution is to the true distribution. A perfect prediction (predicted probability of 1.0 for the correct class) gives a cross-entropy of 0.

Mathematical Formulation

For a single sample with $C$ classes, the categorical cross-entropy loss is:

$$L = -\sum_{i=1}^{C} y_i \cdot \log(\hat{y}_i)$$

where:

• $y_i$ is the true label for class $i$ (1 for the correct class, 0 for all others, in one-hot encoding)
• $\hat{y}_i$ is the predicted probability for class $i$ (output of the softmax layer)

Since $y_i$ is 0 for all classes except the true class, the sum simplifies to:

$$L = -\log(\hat{y}_c)$$

where $c$ is the index of the correct class. This means the loss depends only on the predicted probability assigned to the correct class.

For a batch of $N$ samples, the total loss is averaged:

$$L_{batch} = -\frac{1}{N} \sum_{j=1}^{N} \log(\hat{y}_{j,c_j})$$

Worked Example

Consider classifying handwritten digits (0-9) using the MNIST dataset. There are 10 classes.

True label: digit 3, one-hot encoded as [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]

Predicted probabilities (after softmax): [0.05, 0.02, 0.10, 0.80, 0.01, 0.01, 0.005, 0.003, 0.001, 0.001]

The loss is:

$$L = -(0 \cdot \log(0.05) + 0 \cdot \log(0.02) + 0 \cdot \log(0.10) + 1 \cdot \log(0.80) + \ldots)$$

$$L = -\log(0.80) \approx 0.223$$

If the model had predicted 0.99 for class 3 instead:

$$L = -\log(0.99) \approx 0.010$$

And if it predicted only 0.10:

$$L = -\log(0.10) \approx 2.303$$

The loss grows rapidly as the predicted probability for the correct class drops, heavily penalizing confident wrong predictions.

How Softmax and Cross-Entropy Work Together

The softmax function converts raw model outputs (logits) into a probability distribution:

$$\hat{y}_i = \frac{e^{z_i}}{\sum_{j=1}^{C} e^{z_j}}$$

where $z_i$ is the logit for class $i$. The outputs sum to 1 and are all positive, forming a valid probability distribution.

In practice, Keras combines softmax and cross-entropy into a single numerically stable operation using the from_logits=True parameter:

loss_fn = tf.keras.losses.CategoricalCrossentropy(from_logits=True)

This avoids numerical issues that arise from computing $\log(\text{softmax}(z))$ in two separate steps (where softmax can produce values very close to 0, causing $\log(0) = -\infty$).

Implementation in Keras

Standard usage with softmax output layer:

1import tensorflow as tf
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Dense(128, activation='relu'),
5    tf.keras.layers.Dense(10, activation='softmax')
6])
7
8model.compile(
9    optimizer='adam',
10    loss='categorical_crossentropy',
11    metrics=['accuracy']
12)

Recommended: Using from_logits=True

1model = tf.keras.Sequential([
2    tf.keras.layers.Dense(128, activation='relu'),
3    tf.keras.layers.Dense(10)  # No softmax here
4])
5
6model.compile(
7    optimizer='adam',
8    loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
9    metrics=['accuracy']
10)

This approach is numerically more stable and is recommended by TensorFlow's documentation.

CategoricalCrossentropy vs SparseCategoricalCrossentropy

Use SparseCategoricalCrossentropy when your labels are integers. Use CategoricalCrossentropy when your labels are already one-hot encoded. The mathematical computation is the same either way.

Common Pitfalls

• Forgetting softmax: If your final layer has no activation and you do not set from_logits=True, the loss function will treat raw logits as probabilities, producing incorrect gradients.
• Using with sigmoid: CategoricalCrossentropy expects a probability distribution that sums to 1. Using sigmoid activation (which produces independent probabilities) instead of softmax will give wrong results. For multi-label classification where each class is independent, use BinaryCrossentropy instead.
• Label smoothing: Keras supports label smoothing via CategoricalCrossentropy(label_smoothing=0.1), which replaces hard 0/1 targets with soft targets like 0.01/0.91. This can improve generalization by preventing the model from becoming overconfident.

Summary

Keras's CategoricalCrossentropy computes $L = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)$, which simplifies to $-\log(\hat{y}_c)$ for one-hot labels. It penalizes predictions that assign low probability to the correct class. For best numerical stability, use from_logits=True and omit the softmax activation from the final layer. Choose SparseCategoricalCrossentropy when labels are integers rather than one-hot vectors.