Introduction

When TFBertForSequenceClassification predicts the same label for almost every example, the root cause is usually not the model architecture itself. It is more often a data pipeline problem, label imbalance, training instability, or evaluation code that is decoding the outputs incorrectly.

Check the Data and Labels First

Start by verifying class balance and label encoding. If one class dominates the dataset heavily, the model may learn a trivial majority-class strategy that still looks deceptively accurate.

A quick inspection helps:

1from collections import Counter
2
3labels = [0, 0, 0, 1, 1, 0, 0]
4print(Counter(labels))

Also verify that your labels really match the intended class mapping all the way through tokenization, dataset creation, training, and evaluation. A swapped mapping can make the model look broken when the pipeline is actually misaligned.

Verify Training and Prediction Code

Make sure the model is being trained with sensible hyperparameters and that prediction code is reading logits correctly.

1import tensorflow as tf
2from transformers import AutoTokenizer, TFBertForSequenceClassification
3
4tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
5model = TFBertForSequenceClassification.from_pretrained(
6    "bert-base-uncased",
7    num_labels=2
8)
9
10texts = ["good movie", "bad movie"]
11batch = tokenizer(texts, padding=True, truncation=True, return_tensors="tf")
12
13outputs = model(batch)
14logits = outputs.logits
15predictions = tf.argmax(logits, axis=-1)
16
17print(logits.numpy())
18print(predictions.numpy())

If you skip argmax or read the wrong tensor, you can misinterpret the model output and think it is stuck on one class even when it is not.

Common Training Causes

The usual causes of uniform predictions include:

• severe class imbalance
• learning rate that is too high or too low
• too few training steps
• shuffled or corrupted labels
• truncation that removes the informative part of the text

A safer training setup often includes lower learning rates and validation monitoring:

1optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5)
2loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
3
4model.compile(optimizer=optimizer, loss=loss, metrics=["accuracy"])

If the loss does not move or the validation behavior looks flat, the model may not be learning useful signal at all.

Inspect Predictions Beyond Accuracy

Do not rely on overall accuracy alone. Print a confusion matrix or per-class counts on the validation set.

If every prediction is class 0, you will see it immediately from the prediction distribution:

1import numpy as np
2
3preds = np.array([0, 0, 0, 0, 0])
4print(np.unique(preds, return_counts=True))

That makes debugging faster than trying to infer the issue from a single scalar metric.

Common Pitfalls

The biggest mistake is assuming the model is at fault before verifying the labels and evaluation code. Uniform predictions are often a symptom of a broken training setup rather than a broken BERT model.

Another common issue is class imbalance. If the dataset is skewed and you train briefly, the model can converge to the majority class surprisingly easily.

People also forget that tokenization choices matter. If truncation removes the informative part of each example, the classifier may not have enough signal to distinguish labels.

Finally, inspect raw logits and prediction counts directly. That usually reveals whether the model is truly collapsed or whether the decoding code is the real problem.

Early inspection saves hours later.

Summary

• Uniform predictions usually point to data, label, or training issues before they point to a model bug.
• Verify class balance and label mapping first.
• Confirm that prediction code uses the model logits correctly.
• Use sensible fine-tuning settings such as a small learning rate and validation checks.
• Inspect prediction distributions and confusion-style outputs instead of trusting one aggregate metric.