[KongchangAI]
· 2 min read· 1,147 words

Machine Learning Relay Protection Assessment: Misoperation and Failure-to-Trip Risks Behind High AUC

Machine Learning Relay Protection Assessment: Misoperation and Failure-to-Trip Risks Behind High AUC

PROTECT-90 dataset reveals ML fault detectors with high AUC differ in failure-to-trip rates by two orders of magnitude, requiring protection-oriented evaluation.

The study builds the PROTECT-90 dataset (9,022 simulated fault events on a 90 kV double-circuit line) to conduct a protection-oriented assessment of gradient boosted trees, CNNs, and logistic regression fault detectors. Although all three models achieve AUC-ROC above 0.999, under a security budget of 10^-3 their failure-to-trip rates range from 0.007% to 1.99%, a difference of two orders of magnitude; post-clearing false-trip rates are about 0.12-0.16% per window, and 11.3-19.1% of events contain nuisance assertions. Physical correlation analysis shows failure-to-trip is most consistently correlated with higher fault resistance, and the rank correlation between analytical current proxies and waveform currents is only 0.62. The study emphasizes that aggregate metrics mask the real-world consequences of misoperations and failure-to-trip events, and recommends a protection-oriented evaluation framework.

Introduction

Machine learning (ML) fault detectors have attracted considerable attention in power system protection, but aggregate metrics (such as AUC-ROC) may mask the real-world consequences of misoperations and failure-to-trip events. A study titled "Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems" constructs the PROTECT-90 dataset to conduct a protection-oriented assessment of three ML detectors alongside a sequence current reference. This article interprets its methodology, findings, and implications.

rss source: Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

Evaluation Framework: PROTECT-90 and Protection-Oriented Metrics

The study builds the PROTECT-90 dataset from 9,022 simulated fault events on a 90 kV double-circuit line, designed to emulate realistic protection scenarios. The evaluation framework combines five window conditions, causal window-end labeling, event-grouped cross-validation, and operating point selection under a fixed security budget. Reported metrics cover security, dependability, detection timing, decision margin, and event-level nuisance assertions. This design emphasizes the protection perspective: it focuses not only on classification accuracy but also on the practical impact of misoperations and failure-to-trip events.

rss source: Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

Security budget refers to the upper limit on the misoperation rate allowed when selecting an operating point, typically set to a very small value (e.g., (10^{-3})) to constrain the model's false actions under non-fault conditions. Given a security budget, the decision threshold is adjusted so that the misoperation rate does not exceed that limit, and metrics such as the failure-to-trip rate are then evaluated under this constraint. This approach mimics the preference of protection engineers in real deployments: "first ensure no misoperation, then minimize missed trips as much as possible." The differences in failure-to-trip rates among models under the same security budget reflect differences in how efficiently they learn fault features.

Model Performance: AUC Above 0.999, but Failure-to-Trip Rates Differ Significantly

Gradient boosted trees, convolutional neural networks, and logistic regression all achieve excellent AUC-ROC levels above 0.999. However, at a security budget of $10^{-3}$, their developed-fault missed-trip rates range from 0.007% to 1.99%, a difference of two orders of magnitude. This shows that a high AUC does not directly translate into protection reliability, and that different models exhibit significant differences in failure-to-trip performance under the same security constraint.

rss source: Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

AUC-ROC (area under the receiver operating characteristic curve) is a common metric for measuring the overall discriminative ability of a binary classification model. It ranges from 0 to 1, with values closer to 1 indicating a stronger ability to separate positive and negative samples. However, AUC-ROC is obtained by integrating over all possible classification thresholds, reflecting the model's ranking ability rather than its performance at a specific operating point. In protection scenarios, a fixed threshold must be selected for actual deployment, and only then do the specific proportions of misoperations and failure-to-trip events carry engineering significance. Therefore, two models with AUCs both above 0.999 may exhibit markedly different failure-to-trip rates under the same security budget, which is precisely why the original paper emphasizes that "a high AUC does not directly translate into protection reliability."

Misoperations and Disturbances: Post-Clearing False-Trip Rates and Event-Level Impact

Post-clearing false-trip rates are approximately 0.12-0.16% per window, while 11.3-19.1% of events contain nuisance assertions. This means that even if the model performs well in most windows, unnecessary actions may still occur at the event level, posing a potential risk to the protection system.

rss source: Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

Post-clearing window refers to the time segment after a fault has been cleared. In a protection system, current should quickly return to normal levels after fault clearing; if the model still outputs a fault signal during this stage, it will cause a misoperation. Nuisance assertions refer to the number of undesired alarms the model produces across multiple windows within a single event. Even if these alarms do not lead to a final misoperation, they increase event-level uncertainty. Event-level statistics reflect the behavior of the protection system throughout the complete fault process better than window-level statistics, because a single misoperating window within an event may trigger an unnecessary trip.

Physical Correlation Analysis: High Fault Resistance Strongly Correlated with Failure-to-Trip

Among the physical descriptors analyzed, failure-to-trip is most consistently correlated with higher fault resistance within the sampling range. In addition, analytical current proxies show that fault current gradients are only partially reproduced by waveform-derived currents (rank correlation 0.62). Within the sampling domain, no single topology state or fault phase consistently dominates. These findings suggest that physical factors such as fault resistance may influence model decisions and should be considered in feature engineering and evaluation.

rss source: Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

Fault resistance is the impedance present at the fault point, which limits the magnitude of the fault current and weakens the fault signature. The current waveform of a high-resistance fault differs only slightly from normal load current, making it difficult for conventional overcurrent protection to detect and easy for machine learning-based detectors to miss. Rank correlation measures the agreement between the rankings of two variables; here it is used to compare the analytically computed current gradient with the waveform-derived current gradient. A rank correlation of 0.62 indicates only partial agreement, suggesting that relying solely on waveform features may lose some physical information.

Limitations and Implications: Boundaries of Security Assessment and Future Directions

Because PROTECT-90 contains only fault events, the security assessment is based on pre-fault and post-clearing windows rather than independent non-fault disturbances. This limits the comprehensiveness of the security evaluation. Future research needs to incorporate more non-fault scenarios and explore the model's generalization capability across diverse topologies and fault types. For protection engineers, this study emphasizes that one should go beyond aggregate metrics and adopt a protection-oriented evaluation framework to select and deploy ML detectors.

rss source: Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

One-line Summary

The PROTECT-90 dataset reveals that ML fault detectors with high AUC can differ in failure-to-trip rates by two orders of magnitude, necessitating protection-oriented evaluation.

Paragraph Summary

This study constructs the PROTECT-90 dataset (9,022 simulated fault events on a 90 kV double-circuit line) to conduct a protection-oriented assessment of three machine learning fault detectors: gradient boosted trees, convolutional neural networks, and logistic regression. Although all three models achieve AUC-ROC above 0.999, under a security budget of 10^-3 their failure-to-trip rates range from 0.007% to 1.99%, a difference of two orders of magnitude; post-clearing false-trip rates are approximately 0.12-0.16% per window, and 11.3-19.1% of events contain nuisance assertions. Physical correlation analysis shows that failure-to-trip is most consistently correlated with higher fault resistance, and the rank correlation between analytical current proxies and waveform currents is only 0.62. The study emphasizes that aggregate metrics mask the real-world consequences of misoperations and failure-to-trip events, and recommends adopting a protection-oriented evaluation framework.

#machine learning fault detection#power system protection#PROTECT-90 dataset#AUC-ROC#failure-to-trip rate#misoperation rate#fault resistance#protection-oriented evaluation
Share:

Related articles