Introduction

In TensorFlow and Keras, the simplest ensemble is usually multiple trained models whose predictions are averaged. The modern way to build that is with the Keras Functional API and a merging layer such as Average, which gives you one reusable model object for inference and saving.

Train More Than One Base Model

An ensemble only helps if the base models are not making exactly the same mistakes. One simple way to create diversity is to change width, dropout, or random seed:

1import numpy as np
2import tensorflow as tf
3from tensorflow import keras
4from tensorflow.keras import layers
5
6rng = np.random.default_rng(42)
7X = rng.normal(size=(2000, 10)).astype("float32")
8y = (0.7 * X[:, 0] - 0.3 * X[:, 1] + rng.normal(0, 0.1, 2000)).astype("float32")
9
10X_train, X_val = X[:1600], X[1600:]
11y_train, y_val = y[:1600], y[1600:]
12
13def build_model(width, dropout):
14    inputs = keras.Input(shape=(10,))
15    x = layers.Dense(width, activation="relu")(inputs)
16    x = layers.Dropout(dropout)(x)
17    x = layers.Dense(width // 2, activation="relu")(x)
18    outputs = layers.Dense(1)(x)
19    model = keras.Model(inputs, outputs)
20    model.compile(optimizer="adam", loss="mse", metrics=["mae"])
21    return model
22
23model_a = build_model(64, 0.10)
24model_b = build_model(96, 0.20)
25
26model_a.fit(X_train, y_train, epochs=5, batch_size=32, verbose=0)
27model_b.fit(X_train, y_train, epochs=5, batch_size=32, verbose=0)

The goal is not to make the models wildly different. It is to avoid several identical copies that contribute little ensemble benefit.

Build A Keras Averaging Ensemble

Once the base models are trained, combine them into one graph:

1inputs = keras.Input(shape=(10,))
2pred_a = model_a(inputs)
3pred_b = model_b(inputs)
4avg_pred = layers.Average()([pred_a, pred_b])
5
6ensemble = keras.Model(inputs, avg_pred)
7ensemble.compile(optimizer="adam", loss="mse", metrics=["mae"])
8
9loss, mae = ensemble.evaluate(X_val, y_val, verbose=0)
10print("Ensemble MAE:", round(float(mae), 4))

This uses the Keras Average layer, which is the cleanest way to express element-wise averaging in a model graph.

Because the ensemble is still a Keras model, you can call predict, evaluate, and save on it normally.

Weighted Averaging Is Also Easy

If one model is consistently stronger, a weighted average may perform better:

1inputs = keras.Input(shape=(10,))
2pred_a = model_a(inputs)
3pred_b = model_b(inputs)
4
5weighted_pred = layers.Lambda(lambda xs: 0.7 * xs[0] + 0.3 * xs[1])([pred_a, pred_b])
6weighted_ensemble = keras.Model(inputs, weighted_pred)
7weighted_ensemble.compile(optimizer="adam", loss="mse", metrics=["mae"])

Choose weights based on validation results, not intuition alone.

If the weighted ensemble is not better than plain averaging, prefer the simpler mean.

Classification Usually Averages Probabilities

For classification, the usual pattern is to average predicted probabilities instead of hard labels:

# example idea
# probs = (model1.predict(X) + model2.predict(X) + model3.predict(X)) / 3.0
# labels = (probs >= 0.5).astype("int32")

Averaging hard labels throws away confidence information too early. Probabilities preserve more signal for the ensemble.

Save The Ensemble Like Any Other Model

Because the ensemble is one Keras graph, saving is normal:

ensemble.save("ensemble.keras")
restored = keras.models.load_model("ensemble.keras")
print(restored.predict(X_val[:2], verbose=0).shape)

That is a major advantage over ad hoc Python-side averaging code, which often complicates deployment.

It also keeps the serving path aligned with training-time evaluation, because the same model object that you validated is the one you can export and deploy.

Common Pitfalls

One common mistake is training several base models that are effectively identical and then expecting a large ensemble gain.

Another issue is evaluating the ensemble without comparing it against the best individual model on the same validation set.

A third problem is choosing weighted averages without evidence that the chosen weights improve validation performance.

Finally, some teams build a Python-only averaging wrapper instead of a Keras model graph, which makes saving and serving more awkward than necessary.

Summary

• The easiest TensorFlow ensemble is multiple Keras models combined with an averaging layer.
• Base models should differ enough to make partially different errors.
• Start with plain averaging before trying weighted combinations.
• For classification, average probabilities rather than hard labels.
• Build the ensemble as one Keras model so evaluation, saving, and serving stay simple.