Introduction

In multi-label classification, each sample can belong to several classes at once, so the metric choices are different from ordinary multi-class classification. In Keras, the biggest mistake is treating a multi-label problem like a softmax-based single-label problem and then reading an accuracy number that sounds good but measures the wrong thing.

Start with the Right Output Setup

A typical multi-label model uses one sigmoid output per label rather than a softmax over mutually exclusive classes.

1import tensorflow as tf
2from tensorflow import keras
3
4num_features = 20
5num_labels = 5
6
7model = keras.Sequential([
8    keras.layers.Input(shape=(num_features,)),
9    keras.layers.Dense(32, activation="relu"),
10    keras.layers.Dense(num_labels, activation="sigmoid")
11])

Each output unit predicts the probability of one label independently. That is why a sample can end up with several positive labels at once.

For the same reason, the usual loss is often BinaryCrossentropy, not CategoricalCrossentropy.

Why Plain Accuracy Can Be Misleading

If most labels are zeros, a model can look accurate simply by predicting mostly zeros. In multi-label data, that happens often because positive labels may be sparse.

So although Keras can report an accuracy-like metric, you should ask what it is actually measuring.

In multi-label setups, useful metrics often include:

• binary accuracy
• precision
• recall
• AUC
• custom F1-style metrics

Each one captures a different tradeoff.

A Good Keras Starting Point

A solid baseline compile step looks like this:

1model.compile(
2    optimizer="adam",
3    loss=keras.losses.BinaryCrossentropy(),
4    metrics=[
5        keras.metrics.BinaryAccuracy(name="binary_accuracy"),
6        keras.metrics.Precision(name="precision"),
7        keras.metrics.Recall(name="recall"),
8        keras.metrics.AUC(name="auc", multi_label=True)
9    ]
10)

This already tells you much more than a single generic accuracy value.

BinaryAccuracy checks label-wise correctness after thresholding. Precision and Recall help you understand false positives versus false negatives. AUC can be useful when threshold selection is still in flux.

Thresholds Matter

A multi-label model usually outputs probabilities, not final label decisions. The default threshold is often 0.5, but that may not be the best operating point for your task.

Example prediction post-processing:

1import numpy as np
2
3probs = model.predict(np.random.rand(2, num_features), verbose=0)
4preds = (probs >= 0.5).astype(int)
5
6print(probs)
7print(preds)

If recall matters more than precision, you might lower the threshold. If false positives are costly, you might raise it.

That is another reason to be careful with metrics. The same model can look quite different depending on the threshold used to convert probabilities into labels.

A Custom F1 Metric Is Often Helpful

Keras does not always give you the exact F1 variant you want out of the box, especially for multi-label tasks. A custom metric can make evaluation more aligned with how the model will actually be used.

1import tensorflow as tf
2
3
4def multilabel_f1(y_true, y_pred):
5    y_pred = tf.cast(y_pred >= 0.5, tf.float32)
6    y_true = tf.cast(y_true, tf.float32)
7
8    tp = tf.reduce_sum(y_true * y_pred)
9    fp = tf.reduce_sum((1 - y_true) * y_pred)
10    fn = tf.reduce_sum(y_true * (1 - y_pred))
11
12    precision = tp / (tp + fp + 1e-7)
13    recall = tp / (tp + fn + 1e-7)
14    return 2 * precision * recall / (precision + recall + 1e-7)

Then compile with it:

1model.compile(
2    optimizer="adam",
3    loss="binary_crossentropy",
4    metrics=[multilabel_f1]
5)

In production work, many teams still compute the final reporting metrics outside the training loop with scikit-learn or custom evaluation scripts, because it gives more control over micro, macro, and per-label calculations.

Think About Micro, Macro, and Per-Label Views

A single overall metric can hide important failure modes.

Useful perspectives include:

• micro averaging, which pools all label decisions together
• macro averaging, which treats each label equally
• per-label metrics, which show whether one class is being ignored

If one label is rare but important, macro and per-label reporting will usually tell you more than a single averaged score.

Common Pitfalls

The biggest mistake is using softmax with one-hot assumptions for a truly multi-label problem. Multi-label outputs need independent sigmoid probabilities.

Another common problem is trusting generic accuracy too much. In sparse label settings, a model that predicts mostly zeros can look deceptively strong.

Developers also forget that threshold choice affects evaluation. A metric based on thresholded predictions is partly a metric of your threshold policy, not just of the network weights.

Finally, do not assume one metric is enough. Precision, recall, AUC, and F1 can disagree, and that disagreement is often the exact signal you need.

Summary

• Multi-label models usually use sigmoid outputs and binary cross-entropy loss.
• 'BinaryAccuracy, Precision, Recall, and multi-label AUC are common Keras metrics.'
• Accuracy alone is often misleading for sparse multi-label data.
• Threshold selection strongly affects reported performance.
• Custom or external F1 calculations are often useful for real evaluation.