ECE

Expected Calibration Error (ECE) bins prediction samples by confidence level, computes the weighted average of the difference between predicted confidence and actual accuracy within each bin, and ideally approaches 0.

Model calibration refers to the degree to which a model's output probabilities match its actual accuracy rate; for a well-calibrated model, if the probability it gives for a certain type of question is 0.8, then about 80% of such questions are indeed answered correctly.

Calibration refers to the degree to which a model's expressed confidence matches the actual frequency of correctness, and is commonly quantified using reliability diagrams and Expected Calibration Error (ECE).

In IDP scenarios, ECE/MCE reveal the risk of the model "confidently making mistakes" better than plain accuracy, and teams can use them to set confidence thresholds and route low-confidence results to human review.

After temperature scaling calibration, the ECE dropped to 0.055.

The expected calibration error (ECE) of the initial model was 0.117.

Common metrics for measuring a model's ability to express uncertainty include the calibration curve and ECE (Expected Calibration Error).

On calibration, Jev's ECE in independent tests varies between 0.02 and 0.35 depending on the task; Laya's ECE is 0.466 before temperature fitting (overconfident) and drops to 0.081 after fitting; Drex has not published calibration data.

On 300 practice exam papers labeled by Qwen3-32B, fine-tuned v3 of Laya raised the judgment macro-F1 across 11 questions from 0.33 to 0.57 versus the zero-shot version, cut calibration error from 0.15 to 0.07, and reduced danger-level MAE from 2.78 to 1.56.

Calibration and discrimination are two independent metrics: discrimination measures the model's ability to rank positives ahead of negatives (such as ROC-AUC), while calibration measures whether the model's output probabilities match true frequencies; commonly used calibration measures include Expected Calibration Error (ECE) and reliability diagrams, and correction methods include Platt scaling, isotonic regression and temperature scaling.

The confidence column shipped by Largha-150M-v1 has an ECE of 0.2519, and the built-in recalibration column (temperature scaling) reduces it to 0.0168.

The escalation gate based on the confidence threshold τ must first be recalibrated with temperature scaling, otherwise the threshold is invalid, because the shipped confidence column has an ECE of 0.2519.

ECE (Expected Calibration Error) measures the consistency between a model's confidence and its true accuracy; a lower ECE indicates more trustworthy probability outputs.

Point-estimate confidence fails under distribution shift, whereas Bayesian uncertainty measures such as MC Dropout can genuinely identify signal failure.

The Expected Calibration Error (ECE) increased in seven out of eight event pairs, and in affected scenarios, 12%–51% of the area was confidently mislabeled by the model with near-zero accuracy.

On a test set of 333 articles and 1,218 prompts, Fiorillo v0.5 achieved an Expected Calibration Error (ECE) of 0.0168, well below the preset upper limit of 0.05.

On top of the Brier score, RLCD adds a per-bin calibration penalty term similar to the ECE approach, reducing rewards in overconfident intervals and increasing rewards in underconfident intervals.

The author found that using only a global -λ·ECE penalty across the batch is nearly ineffective, because it is directly absorbed and offset by the PPO or GRPO baseline.

There is an unverified risk with calibration rewards: they may simply flatten the confidence of all outputs toward a middle value, performing well on the ECE metric while losing discriminative information.

Confidence calibration is typically measured using reliability diagrams and Expected Calibration Error (ECE).