UtilityAUC
A metric for evaluating selective prediction that measures the area under the utility-weighted risk-coverage curve. It distinguishes models with similar accuracy but different distributions of high-cost errors.
Timeline (last 90 days)
The researchers introduced the UtilityAUC metric to measure the area under the utility curve determined by the model's response ordering.
UtilityAUC can capture how well a model aligns with utility preferences, not merely error frequency.
Theoretical analysis shows that UtilityAUC can distinguish between models that perform similarly on standard accuracy and AURC.
The research team evaluated 20 state-of-the-art models and found that higher accuracy or more advanced architectures do not always yield higher UtilityAUC.
AURC focuses on error frequency but treats all errors as equivalent, whereas UtilityAUC introduces utility weights so that high-cost errors account for a larger share in the curve integral.
All Facts (5)
The researchers introduced the UtilityAUC metric to measure the area under the utility curve determined by the model's response ordering.
50%UnverifiedUtilityAUC can capture how well a model aligns with utility preferences, not merely error frequency.
50%UnverifiedTheoretical analysis shows that UtilityAUC can distinguish between models that perform similarly on standard accuracy and AURC.
50%UnverifiedThe research team evaluated 20 state-of-the-art models and found that higher accuracy or more advanced architectures do not always yield higher UtilityAUC.
50%UnverifiedAURC focuses on error frequency but treats all errors as equivalent, whereas UtilityAUC introduces utility weights so that high-cost errors account for a larger share in the curve integral.
50%