[KongchangAI]
· 2 min read· 1,058 words

LLM-Enhanced Reinforcement Learning: Human-in-the-Loop Calibration for Real-Time Optimal Power Flow

LLM-Enhanced Reinforcement Learning: Human-in-the-Loop Calibration for Real-Time Optimal Power Flow

LLMs convert semantic operating requirements into reward signals, and combined with human-in-the-loop calibration, improve the safety and stability of RT-OPF reinforcement learning.

To address the difficulty of precisely encoding expert preferences and safety requirements with numerical rewards in deep reinforcement learning for real-time optimal power flow, this paper introduces a DRL framework that integrates large language models with human-in-the-loop calibration. During offline training, the LLM converts semantic operating requirements such as "avoid line overload" and "prioritize critical loads" into sparse language evaluations, which jointly shape the policy with physical rewards; a selective HITL mechanism conducts human review and correction of suspicious LLM judgments to prevent erroneous feedback from continuously contaminating training. Experiments on the IEEE 30-bus and 118-bus systems show that LLM feedback improves policy safety and outperforms the original TD3, while adding HITL calibration yields the best overall performance and more stable training. This paradigm provides a reference path for incorporating expert preferences into reinforcement learning in safety-critical domains.

Real-time optimal power flow (RT-OPF) is a core problem in power system operation, requiring fast, economical, and safe dispatch decisions under constraints on active power, reactive power, voltage, and more. Traditional numerical optimization methods carry heavy computational burdens under time-varying conditions, and while deep reinforcement learning (DRL) can provide fast policies, it faces a key bottleneck: many practical operating preferences and safety requirements are difficult to encode precisely with purely numerical reward functions. A recent paper proposes a DRL framework that integrates large language models (LLMs) with human-in-the-loop (HITL) calibration, offering a new approach to the controllability and reliability of RT-OPF.

Overview of the LLM-enhanced DRL framework

Why RT-OPF Needs Language Feedback

Decisions in real-time optimal power flow involve not only economics but also a large number of semantic operating requirements, such as "avoid sustained overload on a particular line" or "prioritize power supply to critical loads." These preferences often rely on expert experience and are difficult to translate into precise mathematical expressions or scalar rewards. The paper points out that DRL relying solely on numerical rewards tends to learn policies that deviate from actual operating practices, and the training process is highly sensitive to reward design.

The introduction of LLMs is precisely to fill this gap. During the offline training phase, the framework uses an LLM to provide language evaluations of the agent's behavior (e.g., "this action leads to an elevated risk of voltage violations") and converts this sparse semantic feedback into additional reward signals, which jointly shape the policy together with physics-based rewards. In this way, operators can express preferences in natural language without having to repeatedly tune reward weights.

Optimal power flow (OPF) is a classic optimization problem in power system operation, whose goal is to minimize generation cost or network losses while satisfying physical constraints such as nodal power balance, line flow limits, generator output, and voltage magnitudes. Real-time OPF (RT-OPF) requires this optimization to be solved on a rolling basis at minute-level or even shorter timescales to cope with load fluctuations and changes in renewable generation output. Although traditional solvers (e.g., interior-point methods) are highly accurate, each solve requires a full iteration, resulting in a significant computational burden in strongly time-varying scenarios. Deep reinforcement learning fits a policy network as a mapping from system states to dispatch actions through offline training, requiring only a single forward inference online, and thus has millisecond-level response potential; however, its reward function must be scalar, making it difficult to directly express semantic constraints such as "avoid overload" or "prioritize critical loads"—which is precisely the motivation for introducing language feedback.

Human-in-the-Loop Calibration: Adding a Safeguard to LLM Judgments

LLM judgments are not always reliable. The paper particularly emphasizes that language evaluations may be biased or erroneous, and if used continuously for training without screening, they may instead introduce new risks. To this end, the authors design a selective HITL calibration mechanism: after the LLM provides an evaluation, a human (or expert rules) checks the "suspicious" judgments and corrects them, and then the calibrated feedback is used for policy learning.

This design cleverly balances automation and reliability—the LLM provides high-frequency, low-cost semantic signals, while humans intervene only at a few key nodes, avoiding the heavy burden of full manual annotation. The paper describes this mechanism as "reviewing and correcting LLM judgments before they continuously shape the learning process," reflecting a cautious attitude toward safety.

Human-in-the-Loop (HITL) is a system design paradigm that embeds human judgment into automated processes, commonly used in machine learning for data annotation, model evaluation, and safety review. Unlike full manual annotation, selective HITL introduces human intervention only at key nodes where system confidence is low or consequences are irreversible, thereby striking a balance between quality and efficiency. In reinforcement learning scenarios, HITL can be used to correct reward signals, filter dangerous actions, or terminate anomalous training episodes. The selective calibration mechanism in this paper is a concretization of this idea: using LLM semantic evaluations as a high-frequency signal source and triggering human review only for suspicious or high-risk judgments, so that expert effort is concentrated on the few nodes most likely to affect policy safety.

Experimental Validation: IEEE 30-Bus and 118-Bus Systems

The paper conducts case studies on the IEEE 30-bus and IEEE 118-bus systems, comparing three settings: the original TD3 algorithm, TD3 with LLM feedback, and the full framework with further HITL calibration. The results show that LLM feedback can improve policy safety and outperforms the original TD3; after introducing HITL calibration, the overall performance is best and the training behavior is more stable.

This result conveys two messages: language guidance can indeed inject semantic constraints into DRL, but the unreliability of LLMs must be backstopped by human calibration. Only the combination of the two achieves optimal performance in terms of safety and stability.

TD3 (Twin Delayed Deep Deterministic Policy Gradient) is an off-policy deep reinforcement learning algorithm for continuous action spaces, which can be regarded as an improved version of DDPG. It mitigates Q-value overestimation by introducing twin Critic networks and improves training stability through delayed policy updates and target policy smoothing, and is therefore commonly used in continuous dispatch in power systems (e.g., active power output regulation of generators). This paper uses TD3 as the base agent and, under the same state and action spaces, compares the original TD3, TD3 with LLM semantic rewards, and the full framework with further HITL calibration, thereby attributing performance differences to the two factors of language feedback and human calibration rather than changes in the algorithm itself.

Implications for AI Applications in Power Systems

The value of this research extends beyond RT-OPF. It demonstrates a general paradigm: when reinforcement learning tasks involve expert preferences that are difficult to quantify, LLMs can serve as "semantic translators" that convert natural language requirements into training signals, while human-in-the-loop is responsible for quality control. For safety-critical domains such as power systems, this combination of "language guidance + human calibration" is closer to engineering deployment needs than pure end-to-end learning.

Of course, the framework is currently still in the offline training phase, and real-time performance, LLM inference latency, and the response speed of human calibration during online deployment are all questions that need to be answered in the future. Nevertheless, this research provides a noteworthy direction for the controllable application of DRL in real-time optimal power flow.

#LLM#deep reinforcement learning#real-time optimal power flow#human-in-the-loop#TD3#power systems#IEEE 118-bus
Share:

Related articles