Tensorflow 2.2.0 error Predictions must be 0 Condition x y did not hold element-wise while using Bidirectional LSTM layer

Tensorflow

Error Fix

Bidirectional LSTM

Machine Learning

Deep Learning

Tensorflow 2.2.0 error Predictions must be 0 Condition x y did not hold element-wise while using Bidirectional LSTM layer

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Deep Learning TensorFlow & Keras Machine Learning

Introduction

The error "Predictions must be >= 0" or "Condition x >= y did not hold element-wise" in TensorFlow 2.2.0 occurs when the loss function receives prediction values outside the expected range. This typically happens when using binary_crossentropy or categorical_crossentropy with an output activation that does not constrain values to [0, 1]. The fix is to match the output layer's activation function with the loss function: use sigmoid for binary classification and softmax for multi-class classification.

The Error

python

1import tensorflow as tf
2
3# This model causes the error
4model = tf.keras.Sequential([
5    tf.keras.layers.Embedding(10000, 128, input_length=200),
6    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
7    tf.keras.layers.Dense(1)  # No activation — outputs raw logits
8])
9
10model.compile(
11    optimizer='adam',
12    loss='binary_crossentropy',  # Expects values in [0, 1]
13    metrics=['accuracy']
14)
15
16# During training:
17# InvalidArgumentError: Condition x >= y did not hold element-wise:
18# x (predictions) = [[...]]
19# y (zeros) = [[0]...]

The Dense(1) layer with no activation outputs unbounded values (negative to positive infinity). binary_crossentropy expects probabilities between 0 and 1.

Fix 1: Add the Correct Activation Function

Binary Classification

python

1model = tf.keras.Sequential([
2    tf.keras.layers.Embedding(10000, 128, input_length=200),
3    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
4    tf.keras.layers.Dense(64, activation='relu'),
5    tf.keras.layers.Dense(1, activation='sigmoid')  # Outputs [0, 1]
6])
7
8model.compile(
9    optimizer='adam',
10    loss='binary_crossentropy',  # Compatible with sigmoid
11    metrics=['accuracy']
12)

Multi-Class Classification

python

1num_classes = 5
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Embedding(10000, 128, input_length=200),
5    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
6    tf.keras.layers.Dense(64, activation='relu'),
7    tf.keras.layers.Dense(num_classes, activation='softmax')  # Outputs sum to 1
8])
9
10model.compile(
11    optimizer='adam',
12    loss='sparse_categorical_crossentropy',  # Compatible with softmax
13    metrics=['accuracy']
14)

Fix 2: Use from_logits=True

Instead of adding an activation, tell the loss function to handle raw logits:

python

1model = tf.keras.Sequential([
2    tf.keras.layers.Embedding(10000, 128, input_length=200),
3    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
4    tf.keras.layers.Dense(1)  # No activation — raw logits
5])
6
7model.compile(
8    optimizer='adam',
9    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
10    metrics=['accuracy']
11)

Using from_logits=True is numerically more stable because it combines the sigmoid/softmax with the loss computation in a single operation, avoiding log(0) issues.

Fix 3: Check Label Encoding

python

1import numpy as np
2
3# Verify labels are in the correct range
4print(f"Label min: {y_train.min()}, max: {y_train.max()}")
5# Binary: should be 0 or 1
6# Multi-class with sparse_categorical: should be 0 to num_classes-1
7# Multi-class with categorical: should be one-hot encoded
8
9# Fix labels if needed
10# Binary classification
11y_train = (y_train > 0).astype(np.float32)
12
13# Multi-class with one-hot encoding
14y_train_onehot = tf.keras.utils.to_categorical(y_train, num_classes=5)

Activation and Loss Function Matching Table

Task	Output Activation	Loss Function	Label Format
Binary classification	`sigmoid`	`binary_crossentropy`	0 or 1
Binary (from logits)	None	`BinaryCrossentropy(from_logits=True)`	0 or 1
Multi-class	`softmax`	`categorical_crossentropy`	One-hot
Multi-class (sparse)	`softmax`	`sparse_categorical_crossentropy`	Integer
Multi-class (logits)	None	`CategoricalCrossentropy(from_logits=True)`	One-hot
Regression	None or `linear`	`mse`	Continuous

Complete Working Example

python

1import tensorflow as tf
2import numpy as np
3
4# Generate sample data
5vocab_size = 10000
6max_length = 200
7num_samples = 1000
8
9x_train = np.random.randint(0, vocab_size, (num_samples, max_length))
10y_train = np.random.randint(0, 2, (num_samples, 1)).astype(np.float32)
11
12# Build model with correct activation
13model = tf.keras.Sequential([
14    tf.keras.layers.Embedding(vocab_size, 128, input_length=max_length),
15    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
16    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
17    tf.keras.layers.Dropout(0.5),
18    tf.keras.layers.Dense(64, activation='relu'),
19    tf.keras.layers.Dense(1, activation='sigmoid'),
20])
21
22model.compile(
23    optimizer='adam',
24    loss='binary_crossentropy',
25    metrics=['accuracy']
26)
27
28model.fit(x_train, y_train, epochs=5, batch_size=32, validation_split=0.2)

Common Pitfalls

Missing activation on the output layer: The most common cause. Dense(1) without activation='sigmoid' outputs unbounded values. Both binary_crossentropy and categorical_crossentropy expect inputs in [0, 1] unless from_logits=True is set.
Using softmax with binary_crossentropy: For binary classification, softmax on a single output neuron always outputs 1.0 (softmax of a single value is always 1). Use sigmoid for binary classification or use two output neurons with softmax and categorical_crossentropy.
Labels outside the expected range: If labels contain values other than 0/1 for binary classification (e.g., -1/1 or continuous values), the loss computation fails. Verify label ranges with y_train.min() and y_train.max() before training.
NaN in input data causing NaN predictions: NaN values in the input or embedding lookup produce NaN predictions, which violate the >= 0 condition. Check for NaN with np.isnan(x_train).any() and clean the data before training.
Mixing up sparse and non-sparse categorical crossentropy: categorical_crossentropy expects one-hot encoded labels [0, 1, 0]. sparse_categorical_crossentropy expects integer labels 1. Using the wrong one causes shape mismatches or out-of-range errors.

Summary

The error occurs when predictions fall outside the range expected by the loss function
Match activation to loss: sigmoid + binary_crossentropy, softmax + categorical_crossentropy
Use from_logits=True in the loss function to accept raw logits without an activation layer
Verify label encoding: binary labels should be 0/1, multi-class should be one-hot or integer indices
from_logits=True is numerically more stable and preferred in modern TensorFlow code

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Tensorflow 2.2.0 error Predictions must be 0 Condition x y did not hold element-wise while using Bidirectional LSTM layer

ML System Design practice on Codemia

Introduction

The Error

Fix 1: Add the Correct Activation Function

Binary Classification

Multi-Class Classification

Fix 2: Use from_logits=True

Fix 3: Check Label Encoding

Activation and Loss Function Matching Table

Complete Working Example

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

Track what you have practised

ML System Design practice on Codemia