Introduction

Comparing ROC AUC values is not just a matter of subtracting two scores. To decide whether classifier A is really better than classifier B, you need paired predictions on the same test set and a statistical method for the AUC difference, such as DeLong's test, a paired bootstrap confidence interval, or a permutation test.

First compute paired AUCs on the same test set

The most important design rule is that both models must be evaluated on the same observations. If model A and model B are scored on different test sets, a paired AUC comparison is no longer valid.

Here is a simple starting point:

1import numpy as np
2from sklearn.metrics import roc_auc_score
3
4y_true = np.array([0, 1, 0, 1, 1, 0, 0, 1])
5y_score_a = np.array([0.10, 0.90, 0.30, 0.80, 0.75, 0.20, 0.40, 0.88])
6y_score_b = np.array([0.15, 0.82, 0.35, 0.70, 0.68, 0.25, 0.45, 0.80])
7
8auc_a = roc_auc_score(y_true, y_score_a)
9auc_b = roc_auc_score(y_true, y_score_b)
10
11print("AUC A:", auc_a)
12print("AUC B:", auc_b)
13print("Difference:", auc_a - auc_b)

That tells you the point estimate, but not the uncertainty around the difference.

Bootstrap the AUC difference for a confidence interval

A bootstrap is easy to implement and works well in practice. The idea is to resample the test rows with replacement, recompute both AUCs on each bootstrap sample, and store the difference.

1import numpy as np
2from sklearn.metrics import roc_auc_score
3
4def bootstrap_auc_difference(y_true, y_score_a, y_score_b, n_boot=5000, seed=0):
5    rng = np.random.default_rng(seed)
6    n = len(y_true)
7    diffs = []
8
9    for _ in range(n_boot):
10        idx = rng.integers(0, n, n)
11        y_boot = y_true[idx]
12
13        if np.unique(y_boot).size < 2:
14            continue
15
16        auc_a = roc_auc_score(y_boot, y_score_a[idx])
17        auc_b = roc_auc_score(y_boot, y_score_b[idx])
18        diffs.append(auc_a - auc_b)
19
20    diffs = np.asarray(diffs)
21    ci_low, ci_high = np.quantile(diffs, [0.025, 0.975])
22    p_value = 2 * min(np.mean(diffs <= 0), np.mean(diffs >= 0))
23    return diffs.mean(), (ci_low, ci_high), p_value
24
25mean_diff, ci, p = bootstrap_auc_difference(y_true, y_score_a, y_score_b)
26print("Mean diff:", mean_diff)
27print("95% CI:", ci)
28print("Approx p-value:", p)

If the confidence interval for auc_a - auc_b excludes 0, that is evidence that the models differ on the tested data. This bootstrap p-value is an approximation, but it is often good enough when you want a practical answer in Python without implementing DeLong from scratch.

Use a permutation test for a paired p-value

A permutation test is another good option. Under the null hypothesis that the models are equally good, you can randomly swap the two model scores for each observation and see how often the resulting AUC difference is at least as extreme as the observed one.

1import numpy as np
2from sklearn.metrics import roc_auc_score
3
4def permutation_test_auc(y_true, y_score_a, y_score_b, n_perm=5000, seed=0):
5    rng = np.random.default_rng(seed)
6    observed = roc_auc_score(y_true, y_score_a) - roc_auc_score(y_true, y_score_b)
7    count = 0
8
9    for _ in range(n_perm):
10        swap = rng.random(len(y_true)) < 0.5
11        perm_a = np.where(swap, y_score_b, y_score_a)
12        perm_b = np.where(swap, y_score_a, y_score_b)
13        diff = roc_auc_score(y_true, perm_a) - roc_auc_score(y_true, perm_b)
14        if abs(diff) >= abs(observed):
15            count += 1
16
17    return observed, (count + 1) / (n_perm + 1)
18
19observed_diff, p_value = permutation_test_auc(y_true, y_score_a, y_score_b)
20print("Observed diff:", observed_diff)
21print("Permutation p-value:", p_value)

This gives a paired p-value without assuming normality. It is especially attractive when you want a method you can explain clearly and reproduce from basic building blocks.

Where DeLong fits in

DeLong's test is a classic method for comparing correlated ROC AUCs, and it is often the most efficient analytical approach when both models are evaluated on the same cases. The practical issue is that scikit-learn does not ship a DeLong implementation directly, so many teams either use a third-party implementation or rely on bootstrap and permutation procedures instead.

That means the engineering choice is often:

1. Bootstrap for confidence intervals.
2. Permutation test for a robust paired p-value.
3. DeLong if you need the canonical analytical test and are comfortable adding another implementation.

Common Pitfalls

The biggest mistake is comparing AUCs from different test sets as if the scores were paired. Most significance procedures for model comparison assume the same observations were scored by both models.

Another common issue is using hard class predictions instead of probability-like scores. ROC AUC should be computed from continuous decision scores or probabilities, not from already-thresholded labels.

Developers also over-interpret tiny p-values on very large test sets. Statistical significance does not automatically mean the AUC gap is practically meaningful.

Finally, do not ignore class imbalance and data leakage. A beautifully computed p-value does not rescue a flawed evaluation design.

Summary

• Compare ROC AUCs on the same test cases so the comparison is paired.
• Report the point estimate of the AUC difference and an uncertainty measure.
• A bootstrap gives a practical confidence interval for AUC_A - AUC_B.
• A permutation test gives a clean paired p-value with minimal assumptions.
• DeLong is the classic analytical test, but bootstrap and permutation methods are often easier to implement in Python.