Practice Now
Keras
EarlyStopping
custom metrics
deep learning
model training

EarlyStopping is ignoring my custom metrics defined. Keras model

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Deep LearningTensorFlow & KerasMachine Learning

Introduction

Keras EarlyStopping ignores custom metrics when the monitor parameter does not match the exact metric name in the training logs. The metric name used in EarlyStopping(monitor='...') must match the name that appears in model.fit() output — for validation metrics, this is val_ prefixed to the metric name. If the custom metric function is named my_f1_score, the monitor string must be 'val_my_f1_score'. Common causes include typos in the monitor name, not passing validation_data to fit(), and the metric not being added to model.compile(metrics=[...]).

The Problem

python
1import tensorflow as tf
2
3# Custom metric
4def f1_score(y_true, y_pred):
5    y_pred = tf.round(y_pred)
6    tp = tf.reduce_sum(y_true * y_pred)
7    fp = tf.reduce_sum((1 - y_true) * y_pred)
8    fn = tf.reduce_sum(y_true * (1 - y_pred))
9    precision = tp / (tp + fp + tf.keras.backend.epsilon())
10    recall = tp / (tp + fn + tf.keras.backend.epsilon())
11    return 2 * precision * recall / (precision + recall + tf.keras.backend.epsilon())
12
13model = tf.keras.Sequential([
14    tf.keras.layers.Dense(64, activation='relu'),
15    tf.keras.layers.Dense(1, activation='sigmoid')
16])
17
18model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[f1_score])
19
20# WRONG — monitor name does not match
21early_stop = tf.keras.callbacks.EarlyStopping(
22    monitor='val_f1',  # Wrong name!
23    patience=5
24)
25
26model.fit(x_train, y_train, validation_data=(x_val, y_val),
27          epochs=50, callbacks=[early_stop])
28# WARNING: EarlyStopping conditioned on metric `val_f1` which is not available.
29# Available metrics are: loss, f1_score, val_loss, val_f1_score

The Fix: Match the Exact Metric Name

python
1# Check what names Keras uses
2model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[f1_score])
3
4# The metric name is the function name: "f1_score"
5# For validation: "val_f1_score"
6
7early_stop = tf.keras.callbacks.EarlyStopping(
8    monitor='val_f1_score',  # Matches the function name
9    patience=5,
10    mode='max'  # f1_score should be maximized
11)
12
13model.fit(x_train, y_train, validation_data=(x_val, y_val),
14          epochs=50, callbacks=[early_stop])

Finding the Correct Metric Name

python
1# Method 1: Check after one epoch of training
2history = model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=1)
3print(history.history.keys())
4# dict_keys(['loss', 'f1_score', 'val_loss', 'val_f1_score'])
5
6# Method 2: Check model.metrics_names
7model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[f1_score])
8print(model.metrics_names)
9# ['loss', 'f1_score']
10# Validation versions are prefixed with 'val_'

Custom Metric as a Class

Using a class gives you explicit control over the name:

python
1class F1Score(tf.keras.metrics.Metric):
2    def __init__(self, name='f1_score', **kwargs):
3        super().__init__(name=name, **kwargs)
4        self.tp = self.add_weight(name='tp', initializer='zeros')
5        self.fp = self.add_weight(name='fp', initializer='zeros')
6        self.fn = self.add_weight(name='fn', initializer='zeros')
7
8    def update_state(self, y_true, y_pred, sample_weight=None):
9        y_pred = tf.round(y_pred)
10        y_true = tf.cast(y_true, tf.float32)
11        self.tp.assign_add(tf.reduce_sum(y_true * y_pred))
12        self.fp.assign_add(tf.reduce_sum((1 - y_true) * y_pred))
13        self.fn.assign_add(tf.reduce_sum(y_true * (1 - y_pred)))
14
15    def result(self):
16        precision = self.tp / (self.tp + self.fp + tf.keras.backend.epsilon())
17        recall = self.tp / (self.tp + self.fn + tf.keras.backend.epsilon())
18        return 2 * precision * recall / (precision + recall + tf.keras.backend.epsilon())
19
20    def reset_state(self):
21        self.tp.assign(0)
22        self.fp.assign(0)
23        self.fn.assign(0)
24
25model.compile(optimizer='adam', loss='binary_crossentropy',
26              metrics=[F1Score()])
27
28# Name is controlled: 'f1_score', validation: 'val_f1_score'
29early_stop = tf.keras.callbacks.EarlyStopping(
30    monitor='val_f1_score',
31    patience=5,
32    mode='max'
33)

Setting the Correct Mode

python
1# mode='min' — stop when metric stops decreasing (loss)
2# mode='max' — stop when metric stops increasing (accuracy, f1, AUC)
3# mode='auto' — infers from metric name (risky for custom metrics)
4
5# WRONG — auto may not detect that custom f1 should be maximized
6early_stop = tf.keras.callbacks.EarlyStopping(
7    monitor='val_f1_score',
8    patience=5,
9    mode='auto'  # May incorrectly use 'min'
10)
11
12# CORRECT — explicitly set mode
13early_stop = tf.keras.callbacks.EarlyStopping(
14    monitor='val_f1_score',
15    patience=5,
16    mode='max'  # F1 score should be maximized
17)
18
19# With ModelCheckpoint too
20checkpoint = tf.keras.callbacks.ModelCheckpoint(
21    'best_model.keras',
22    monitor='val_f1_score',
23    mode='max',
24    save_best_only=True
25)

Combining Multiple Callbacks

python
1callbacks = [
2    tf.keras.callbacks.EarlyStopping(
3        monitor='val_f1_score',
4        patience=10,
5        mode='max',
6        restore_best_weights=True  # Restore weights from best epoch
7    ),
8    tf.keras.callbacks.ModelCheckpoint(
9        'best_model.keras',
10        monitor='val_f1_score',
11        mode='max',
12        save_best_only=True
13    ),
14    tf.keras.callbacks.ReduceLROnPlateau(
15        monitor='val_f1_score',
16        mode='max',
17        factor=0.5,
18        patience=5
19    )
20]
21
22model.fit(x_train, y_train,
23          validation_data=(x_val, y_val),
24          epochs=100,
25          callbacks=callbacks)

Common Pitfalls

  • Typo in the monitor string: monitor='val_f1' does not match 'val_f1_score'. Keras prints a warning but continues training without early stopping. Always run one epoch first and check history.history.keys() to see the exact metric names available.
  • Forgetting validation_data in model.fit(): Without validation data, the val_ prefixed metrics do not exist. monitor='val_f1_score' triggers the "metric not available" warning. Either pass validation_data=(x_val, y_val) or use validation_split=0.2.
  • Using mode='auto' with custom metrics: Auto mode infers direction from the metric name — it recognizes 'loss', 'acc', 'accuracy' but not custom names. For custom metrics, always set mode='max' or mode='min' explicitly to avoid the wrong stopping direction.
  • Custom function metric vs class metric state: A function metric (plain def f1_score(y_true, y_pred)) is computed per-batch and averaged. A class metric (tf.keras.metrics.Metric) accumulates state across batches. For metrics like F1 that depend on global counts (TP, FP, FN), the class version is mathematically correct while the function version gives a biased average.
  • Not adding the metric to model.compile(metrics=[]): If the custom metric is not passed to compile, it does not appear in training logs. EarlyStopping(monitor='val_my_metric') cannot find it. Ensure the metric is listed in model.compile(metrics=[my_metric]).

Summary

  • The monitor string must exactly match the metric name in training logs — check with history.history.keys()
  • For validation metrics, prefix with val_: function f1_score becomes monitor='val_f1_score'
  • Always set mode='max' or mode='min' explicitly for custom metrics — mode='auto' may guess wrong
  • Use tf.keras.metrics.Metric subclass for metrics that need per-epoch aggregation (F1, precision, recall)
  • Pass validation_data to model.fit() — without it, val_* metrics do not exist
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.