Temperature Scaling

Calibrated confidence means that when the system gives a score of 0.9, the probability of being correct should be close to 90%.

Temperature scaling is a calibration method that attempts to calibrate logits into more reliable probability estimates.

Routing scenarios typically rely on probabilistic scores for hard-threshold decisions, but neural network softmax outputs often suffer from poor calibration, so in practice post-calibration methods such as temperature scaling and histogram binning are needed, along with ECE evaluation.

Post-hoc temperature scaling calibration was adopted, with an optimal temperature parameter of T=2.12.

The study shows that simple calibration techniques can make automated diatom identification more practical and robust.

Recency bias refers to the model repeatedly encountering the latest task data at the end of training, causing the decision boundary and output confidence to skew toward recent tasks, rendering threshold-based OOD discrimination ineffective, and this cannot be corrected by simple temperature scaling or post-processing.

Calibration and discrimination are two independent metrics: discrimination measures the model's ability to rank positives ahead of negatives (such as ROC-AUC), while calibration measures whether the model's output probabilities match true frequencies; commonly used calibration measures include Expected Calibration Error (ECE) and reliability diagrams, and correction methods include Platt scaling, isotonic regression and temperature scaling.

The calibration toolchain provided by the system includes three methods: expected calibration error (ECE), Brier score and temperature scaling.

The escalation gate based on the confidence threshold τ must first be recalibrated with temperature scaling, otherwise the threshold is invalid, because the shipped confidence column has an ECE of 0.2519.

The confidence column shipped by Largha-150M-v1 has an ECE of 0.2519, and the built-in recalibration column (temperature scaling) reduces it to 0.0168.

概率校准的常见方法包括温度缩放（temperature scaling）、Platt scaling和等渗回归（isotonic regression），需要在一批带标注的真实数据上拟合校准参数。

Temperature scaling calibrates a model by dividing logit values by a learnable scalar parameter T before the softmax; T greater than 1 softens the probability distribution, T less than 1 sharpens the distribution, and the main model does not need to be retrained.

Engineering mitigations for confidence calibration include post-calibrating logprobs via temperature scaling, introducing an independent validation model for scoring, or measuring result consistency through multiple sampling runs.

Technical methods for improving calibration include post-processing approaches such as temperature scaling and isotonic calibration.

In practice, post-processing methods such as temperature scaling and Platt scaling are commonly used to recalibrate confidence scores.

Technical approaches to addressing poor model calibration include post-processing calibration methods such as temperature scaling and Platt Scaling, as well as uncertainty estimation techniques such as Bayesian neural networks and MC Dropout.

多数LLM的自报置信度存在过度自信问题，使用前需进行温度缩放等校准处理

Softmax输出的概率值并不等同于真实的贝叶斯后验概率，而是被过度优化以拉开类别间距，导致预测概率系统性偏高