Introduction

In H2O AutoML, cross-validation AUC and holdout-set AUC are different metrics produced in different ways. Cross-validation AUC summarizes performance from the training-time folds, while holdout AUC comes from scoring a separate dataset that was never used during training. You usually need both, but you retrieve them from different places.

Cross-Validation AUC Belongs to the Trained Model

When AutoML trains a model with cross-validation enabled, each model stores cross-validation metrics. That means the leader model can report its own cross-validated AUC directly.

1import h2o
2from h2o.automl import H2OAutoML
3
4h2o.init()
5
6train = h2o.import_file("train.csv")
7target = "label"
8features = [c for c in train.columns if c != target]
9
10aml = H2OAutoML(max_models=10, seed=42, nfolds=5)
11aml.train(x=features, y=target, training_frame=train)
12
13leader = aml.leader
14print("Cross-validation AUC:", leader.auc(xval=True))

That value comes from the model's internal cross-validation results, not from any external holdout frame.

Holdout AUC Must Be Scored Explicitly

If you have a separate holdout dataset, score the trained model on that dataset and ask for the performance there.

1holdout = h2o.import_file("holdout.csv")
2
3perf = leader.model_performance(holdout)
4print("Holdout AUC:", perf.auc())

This gives you the AUC on truly unseen data. It is often the more important number for final model selection because it reflects actual generalization outside the training folds.

Do Not Confuse the Two Numbers

A common misunderstanding is expecting a "cross-validation AUC for the holdout dataset." That phrase mixes two different evaluation ideas:

• cross-validation AUC is estimated during training from folds carved out of the training frame
• holdout AUC is computed afterward on a separate frame

The holdout frame is not part of the cross-validation process unless you manually build a different evaluation design around it.

Using a Leaderboard Frame

AutoML also lets you provide a leaderboard_frame. That frame is used for ranking models externally rather than relying only on cross-validation or validation metrics.

1aml = H2OAutoML(
2    max_models=10,
3    seed=42,
4    nfolds=5,
5    leaderboard_frame=holdout,
6)
7aml.train(x=features, y=target, training_frame=train)

This can be useful when you want the leaderboard sorted by holdout performance. Even then, the model still keeps its own cross-validation metrics separately. The leaderboard frame does not turn holdout AUC into a cross-validation metric; it just provides another evaluation basis.

Inspecting More Than the Leader

If you want AUC values for several AutoML models, iterate over the leaderboard entries and query each model.

1leaderboard = aml.leaderboard.as_data_frame()
2
3for model_id in leaderboard["model_id"].head(3):
4    model = h2o.get_model(model_id)
5    print(model_id, "xval AUC:", model.auc(xval=True))
6    print(model_id, "holdout AUC:", model.model_performance(holdout).auc())

This is helpful when you want to compare whether the top-ranked cross-validation model is also the strongest on the external holdout data.

Common Pitfalls

The biggest pitfall is assuming cross-validation AUC and holdout AUC are interchangeable. They answer related but distinct questions, and one does not automatically replace the other.

Another common mistake is reading leaderboard metrics without confirming which frame they came from. In H2O AutoML, leaderboard ranking can depend on cross-validation, validation, or a provided leaderboard frame.

Developers also sometimes skip explicit holdout scoring and rely only on cross-validation. That can be acceptable in some workflows, but it does not give you the same level of external validation as a truly untouched holdout set.

Summary

• Use model.auc(xval=True) to retrieve a model's cross-validation AUC.
• Use model.model_performance(holdout).auc() to get AUC on a separate holdout dataset.
• Cross-validation metrics and holdout metrics are different evaluations and should not be merged conceptually.
• A leaderboard_frame changes how models are ranked, but it does not turn holdout performance into cross-validation performance.
• For careful comparison, inspect both metrics on the leader and on other top AutoML models.