Calibration

Calibration means that the probabilities output by the model should match reality; neural networks generally have an overconfidence tendency, so model probabilities should not be blindly trusted, and thresholds should be set only after validation with labeled data.

A model with slightly lower accuracy but well-calibrated confidence is often more suitable for production deployment than a model with higher accuracy that is blindly overconfident.

Calibrated confidence means that when the system gives a score of 0.9, the probability of being correct should be close to 90%.

Model calibration is defined in statistics as: if a model claims a certain answer has 70% confidence, then the actual proportion of correct answers among a large number of such claims should be close to 70%.

Alignment training optimizes the model's statistical behavior in known scenarios rather than establishing formally verifiable security boundaries, and may fail when faced with rare instruction combinations or role-playing frameworks outside the training distribution.

Calibrated confidence refers to the consistency between the confidence value output by a model and the actual accuracy rate; a well-calibrated model that claims 80% confidence should have an actual accuracy rate close to 80%.

Calibration refers to the consistency between the probability estimates a model produces and the actual frequency of occurrence, and is commonly quantified using the Brier Score and calibration curves.

Current mainstream large language models exhibit significant deficiencies in both confidence calibration and selective abstention, tend to be overconfident, and when faced with questions outside the training distribution, tend to produce fluent but incorrect answers.

Current mainstream large language models are generally overconfident, partly because the training objective optimizes for the fluency of predicting the next word rather than the actual reliability of the output.

When AI produces incorrect answers, the problem usually lies in the vectorization method or retrieval parameters rather than the model itself.

The confidence output of large language models is fundamentally a product of probability distributions rather than a genuine judgment of factual truth or falsehood, and may express high confidence in incorrect statements.

In a well-calibrated model, an output score of 0.8 should statistically correspond to an actual accuracy of 80%.

Model calibration refers to the degree to which a model's output probabilities match its actual accuracy rate; for a well-calibrated model, if the probability it gives for a certain type of question is 0.8, then about 80% of such questions are indeed answered correctly.

Large language models commonly suffer from overconfidence and require prompt engineering and calibration strategies to calibrate confidence levels.

Calibration concerns whether the probability values are trustworthy, while discriminative power concerns whether high-confidence samples are more likely to be correct than low-confidence samples; even if calibration is biased, as long as confidence maintains ranking discriminative power, it can still be used as an auxiliary variable for stratified sampling.

Being well calibrated means the confidence the model claims matches its actual accuracy.

Thresholds for score questions are usually chosen at the optimal point based on the business cost curve (such as the cost ratio of false blocking versus missed blocking) rather than simply taking the median; score results often need calibration so that scores can be interpreted as probabilities or stable quantiles.

Demographic parity, equal opportunity, and predictive parity often cannot be mathematically satisfied simultaneously (unless the true default rates are the same across groups and the model is perfect); forcing demographic parity distorts predicted probabilities and harms calibration, while pursuing predictive parity may accept group differences in approval rates.

Accuracy alone cannot support gating thresholds; the decision model must be well calibrated so that the output probability values reflect true occurrence rates.

Confidence self-reported by large models is usually uncalibrated and cannot be directly read as a probability of risk; it can only be used as a ranking signal.