Understanding Keras: Obtaining Unnormalized Logits Instead of Probabilities

Keras, a prominent high-level neural network API written in Python, simplifies the complexity of building deep learning models by providing intuitive APIs. Typically, when implementing classification models using Keras, the final layer outputs probabilities of each class, which is the result of applying the softmax or sigmoid function on the logits produced by the preceding layer. However, there are situations where the raw unnormalized logits are more useful than probabilities. This article delves into these scenarios, the reasons for preferring logits, and how to obtain them from a Keras model.

What Are Logits?

Logits represent the raw predictions generated by a neural network model before they are normalized into probabilities via an activation function such as the softmax function. They are the output of the final linear operation preceding any activation function applied at the model's output layer.

Why Obtain Unnormalized Logits?

There are several instances where logits are preferable over probabilities:

• Numerical Stability in `Loss` Functions: Some loss functions, like `categorical_crossentropy`, work directly on logits as it’s numerically more stable compared to applying loss functions on probabilities.
• Custom `Loss` Functions: When designing complex custom loss functions, having direct access to logits can provide improved control over the computation.
• Advanced Use-Cases: Some advanced techniques, such as distillation in model compression or certain adversarial training methods, require logits.

Obtaining Logits in Keras

To work with logits in Keras, one typically needs to avoid adding an activation function on the final layer of the model. Below are steps and explanations on how to achieve this.

Building a Model to Output Logits

Here's a simple example using the Keras `Sequential` API to construct a model that outputs raw logits:

• Numerical Precision: As logits can have both positive and negative values, careful attention is required to maintain numerical precision.
• Activation Post-processing: If you need probabilities at any point, remember you can always apply activations externally, maintaining flexibility.