Conformal Prediction

The case A boundary trained with only 80 simulations correctly labeled 64 out of 65 independent runs.

The coverage guarantee provided by conformal prediction does not depend on whether the model is well calibrated, making it more robust than softmax probabilities.

Existing uncertainty estimation schemes usually require multiple samplings (5x or more compute), the introduction of a judge model, or reliance on labeled datasets, making them costly and only giving general confidence.

Conformal prediction can be applied as a wrapper layer on top of any already-trained model without requiring retraining or modification of the model architecture.

In scenarios involving time series, online learning, or concept drift, exchangeability is violated, and extended methods such as weighted conformal prediction or adaptive conformal inference are needed to restore coverage guarantees.

Conformal prediction is an uncertainty quantification framework that outputs prediction sets or prediction intervals with rigorous statistical coverage guarantees.

The efficiency of a prediction set is typically measured by the average size of the prediction set; the smaller the set, the better, provided the coverage rate is satisfied.

The size of a prediction set is a useful signal: a larger set indicates greater model uncertainty, while a smaller set or one containing a single label indicates high model confidence.

Conformal prediction relies on two key concepts: nonconformity scores and a calibration set, where the calibration set is data with true labels that is independent of the training set.

The distribution-free nature of conformal prediction requires only the exchangeability assumption, meaning the order of data points does not affect their joint distribution.

Conformal prediction can provide theoretically guaranteed coverage rates under finite samples without relying on the underlying model type (random forests, neural networks, or other black-box models).