Introduction

When a hand-written cross-entropy function behaves strangely, the issue is usually not the formula itself. The usual problems are feeding it the wrong inputs, missing numerical stability safeguards, or mixing up binary cross-entropy with multiclass cross-entropy.

The fastest way to debug it is to ask three questions: are these probabilities or logits, is the target format correct, and can any log(0) happen? Most broken implementations fail one of those checks.

Start With the Correct Formula

For binary classification, cross-entropy for one example is:

- (y * log(p) + (1 - y) * log(1 - p))

where:

• 'y is 0 or 1'
• 'p is the predicted probability of class 1'

A simple NumPy implementation looks like this:

1import numpy as np
2
3def binary_cross_entropy(y_true, y_pred, eps=1e-12):
4    y_true = np.asarray(y_true, dtype=np.float64)
5    y_pred = np.asarray(y_pred, dtype=np.float64)
6    y_pred = np.clip(y_pred, eps, 1 - eps)
7    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
8
9
10y_true = np.array([1, 0, 1, 1])
11y_pred = np.array([0.9, 0.2, 0.8, 0.7])
12
13print(binary_cross_entropy(y_true, y_pred))

If your code differs substantially from that structure, there is a good chance the implementation bug is in the math rather than elsewhere in the training loop.

Logits and Probabilities Are Not the Same Input

One of the most common mistakes is feeding raw logits into a formula that expects probabilities. A logit can be any real number, but log(p) expects a value strictly between 0 and 1.

For example, this is wrong:

logits = np.array([2.3, -1.1, 0.7])
# Wrong: logits are not probabilities
np.log(logits)

If your model outputs logits, convert them first:

• use sigmoid for binary classification
• use softmax for multiclass classification

Or better, use a framework loss function that accepts logits directly and handles the stable transformation internally.

Numerical Stability Matters

Even if you pass probabilities, exact 0 or 1 values cause trouble:

• 'log(0) is undefined'
• values extremely close to 0 can explode the loss

That is why clipping or an equivalent stable formulation is standard practice. In the example above, np.clip prevents y_pred from reaching invalid boundaries.

This is also why framework losses are often safer than manual formulas. Functions such as TensorFlow's sigmoid cross-entropy with logits or PyTorch's cross-entropy loss combine the nonlinear transformation with the loss in a numerically stable way.

Match the Target Format to the Loss

Another common bug is using the wrong target encoding. Binary cross-entropy expects scalar probabilities for a binary target. Multiclass cross-entropy expects either:

• one-hot targets with class probabilities
• or integer class indices, depending on the framework API

A simple multiclass version with one-hot targets looks like this:

1import numpy as np
2
3def multiclass_cross_entropy(y_true, y_pred, eps=1e-12):
4    y_true = np.asarray(y_true, dtype=np.float64)
5    y_pred = np.asarray(y_pred, dtype=np.float64)
6    y_pred = np.clip(y_pred, eps, 1 - eps)
7    return -np.mean(np.sum(y_true * np.log(y_pred), axis=1))
8
9
10y_true = np.array([[1, 0, 0], [0, 1, 0]])
11y_pred = np.array([[0.8, 0.1, 0.1], [0.2, 0.7, 0.1]])
12
13print(multiclass_cross_entropy(y_true, y_pred))

If y_true has shape (batch,) but your formula expects one-hot rows, the result will be wrong even if the code runs.

Trust the Reference Implementation

When debugging, compare your manual function against a well-tested library on the same small input. If the numbers differ, use that minimal example to isolate the bug before touching the model code.

For many projects, the best answer is not "fix the handwritten loss forever" but "use the framework's implementation and move on." Manual loss code is useful for learning and debugging, but production training loops rarely need a custom cross-entropy from scratch.

Common Pitfalls

• Passing logits into a formula that expects probabilities.
• Forgetting the negative sign, which flips the optimization objective.
• Allowing 0 or 1 probabilities and triggering unstable logs.
• Mixing binary and multiclass formulas.
• Using target tensors whose shape or encoding does not match the loss function.

Summary

• Most broken cross-entropy implementations fail because of input type, target format, or numerical stability.
• Know whether your model output is logits or probabilities.
• Clip probabilities or use a stable logits-based framework loss.
• Match binary targets to binary cross-entropy and multiclass targets to multiclass cross-entropy.
• When in doubt, compare against a trusted library implementation on a tiny test case.