Introduction

There is no single built-in “confidence interval for the whole precision-recall curve” in the way many people expect. In practice, a useful answer is to bootstrap the evaluation set, recompute the precision-recall curve many times, and then form percentile bands on a shared recall grid.

Why Precision-Recall Curves Need Extra Care

A precision-recall curve is built by sweeping decision thresholds over model scores. Each threshold gives a precision and recall pair, so the curve reflects model ranking quality rather than a single hard prediction rule.

Uncertainty is harder here than for a single scalar metric because the curve is a function, not one number. That means you need to decide what you want confidence intervals for:

• average precision as a single summary metric
• precision at chosen recall levels
• an approximate confidence band for the whole curve

For most applied workflows, bootstrap resampling is the practical approach.

Bootstrap Strategy

The idea is straightforward:

1. sample the test set with replacement
2. compute the precision-recall curve on that bootstrap sample
3. interpolate precision values onto a common recall grid
4. repeat many times
5. take percentiles across runs

That produces an empirical uncertainty band.

A Runnable Python Example

This example uses scikit-learn and NumPy. It assumes you already have the true labels and the model scores for the positive class.

1import numpy as np
2import matplotlib.pyplot as plt
3from sklearn.metrics import precision_recall_curve
4
5rng = np.random.default_rng(42)
6
7y_true = np.array([0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1])
8y_score = np.array([0.10, 0.92, 0.30, 0.85, 0.73, 0.20, 0.40, 0.88, 0.15, 0.67, 0.35, 0.79])
9
10recall_grid = np.linspace(0.0, 1.0, 100)
11boot_precisions = []
12
13for _ in range(1000):
14    indices = rng.integers(0, len(y_true), len(y_true))
15    y_boot = y_true[indices]
16    s_boot = y_score[indices]
17
18    # Skip degenerate samples that contain only one class.
19    if len(np.unique(y_boot)) < 2:
20        continue
21
22    precision, recall, _ = precision_recall_curve(y_boot, s_boot)
23
24    order = np.argsort(recall)
25    recall_sorted = recall[order]
26    precision_sorted = precision[order]
27
28    interp_precision = np.interp(recall_grid, recall_sorted, precision_sorted)
29    boot_precisions.append(interp_precision)
30
31boot_precisions = np.array(boot_precisions)
32
33lower = np.percentile(boot_precisions, 2.5, axis=0)
34upper = np.percentile(boot_precisions, 97.5, axis=0)
35median = np.median(boot_precisions, axis=0)
36
37plt.figure(figsize=(8, 5))
38plt.plot(recall_grid, median, label="Median bootstrap PR")
39plt.fill_between(recall_grid, lower, upper, alpha=0.25, label="95% bootstrap band")
40plt.xlabel("Recall")
41plt.ylabel("Precision")
42plt.legend()
43plt.show()

This gives an approximate 95% confidence band over recall values from 0 to 1.

Why Interpolation Is Needed

Each bootstrap run produces threshold points at different recall locations. You cannot take percentiles point-by-point unless all curves are aligned to the same x-axis values. Interpolating each curve onto a shared recall grid solves that alignment problem.

This is an approximation, but it is usually good enough for model comparison and reporting.

Confidence Interval for Average Precision

If you only need uncertainty for one summary number, bootstrap average precision instead of the full curve. That is often easier to explain in reports.

1from sklearn.metrics import average_precision_score
2
3scores = []
4for _ in range(1000):
5    indices = rng.integers(0, len(y_true), len(y_true))
6    y_boot = y_true[indices]
7    s_boot = y_score[indices]
8    if len(np.unique(y_boot)) < 2:
9        continue
10    scores.append(average_precision_score(y_boot, s_boot))
11
12print(np.percentile(scores, [2.5, 50, 97.5]))

That gives a confidence interval for a scalar summary rather than a band for the curve.

Common Pitfalls

One common mistake is bootstrapping the training set instead of the held-out evaluation set. The confidence interval should reflect uncertainty in evaluation, not retraining noise, unless you are explicitly studying full pipeline variability.

Another issue is forgetting class balance. Small or highly imbalanced datasets can produce bootstrap samples with only one class, which makes the precision-recall curve undefined or uninformative. Those samples should usually be skipped.

It is also important to remember that the resulting band is an empirical bootstrap approximation, not an exact analytical confidence region.

Summary

• A practical confidence interval for a precision-recall curve is usually built with bootstrap resampling.
• Recompute the curve on each bootstrap sample and interpolate onto a shared recall grid.
• Use percentile bands, such as 2.5% and 97.5%, to form an approximate 95% interval.
• For simpler reporting, bootstrap average precision instead of the full curve.
• Be careful with imbalanced data and bootstrap samples that contain only one class.