AUROC
A common metric for evaluating the ranking ability of binary classifiers, ranging from 0.5 to 1.0, representing the probability that a positive sample receives a higher score than a randomly chosen negative sample. It is often used in monitoring scenarios to rank failed and successful trajectories.
Core Facts
Calibration means that the probabilities output by the model should match reality; neural networks generally have an overconfidence tendency, so model probabilities should not be blindly trusted, and thresholds should be set only after validation with labeled data.
AUC值域为0.5到1.0，0.5代表随机猜测、1.0代表完美预测，不受样本类别不平衡影响
Timeline (last 90 days)
Judge disagreement predicts errors, with AUROC up to 0.80, but the variance inflation from inverse probability weighting exceeds the signal gain.
In machine learning, the equivalence margin can be set based on the practical meaning of the metric, for example, an AUROC difference of no more than 0.01.
To demonstrate equivalence at 0.01 AUROC with 80% power, twelve runs per method are required even if the two methods are exactly equal.
For composite statistics, the study uses a two-level bootstrap method for inference, which does not rely on normality assumptions and is applicable to metrics with non-Gaussian distributions such as AUROC.
The authors used an LLM judge to identify suspicious reports and obtained an AUROC of 0.48 (no better than random), whereas a simple TF-IDF word-frequency classifier achieved an AUROC of 0.65.
Research on cross-model verification shows that the AUROC for blind spot detection is 0.70, while same-model self-checking is only 0.59.
Calibration means that the probabilities output by the model should match reality; neural networks generally have an overconfidence tendency, so model probabilities should not be blindly trusted, and thresholds should be set only after validation with labeled data.
S3KG achieves a peak AUROC of 0.973.
Different cohort construction approaches mean that even if two studies report similar AUROCs, they do not necessarily address the same clinical problem, and a model from one institution cannot be directly deployed at another.
All Facts (11)
Calibration means that the probabilities output by the model should match reality; neural networks generally have an overconfidence tendency, so model probabilities should not be blindly trusted, and thresholds should be set only after validation with labeled data.
75%VerifiedAUC值域为0.5到1.0，0.5代表随机猜测、1.0代表完美预测，不受样本类别不平衡影响
75%UnverifiedJudge disagreement predicts errors, with AUROC up to 0.80, but the variance inflation from inverse probability weighting exceeds the signal gain.
50%UnverifiedTo demonstrate equivalence at 0.01 AUROC with 80% power, twelve runs per method are required even if the two methods are exactly equal.
50%UnverifiedIn machine learning, the equivalence margin can be set based on the practical meaning of the metric, for example, an AUROC difference of no more than 0.01.
50%UnverifiedFor composite statistics, the study uses a two-level bootstrap method for inference, which does not rely on normality assumptions and is applicable to metrics with non-Gaussian distributions such as AUROC.
50%UnverifiedThe authors used an LLM judge to identify suspicious reports and obtained an AUROC of 0.48 (no better than random), whereas a simple TF-IDF word-frequency classifier achieved an AUROC of 0.65.
50%UnverifiedResearch on cross-model verification shows that the AUROC for blind spot detection is 0.70, while same-model self-checking is only 0.59.
50%UnverifiedS3KG achieves a peak AUROC of 0.973.
50%UnverifiedDifferent cohort construction approaches mean that even if two studies report similar AUROCs, they do not necessarily address the same clinical problem, and a model from one institution cannot be directly deployed at another.
50%UnverifiedAUC是衡量二分类模型综合性能的核心指标，0.5代表随机猜测，1.0代表完美分类
50%