Offline Reinforcement Learning

In offline reinforcement learning, unmeasured confounding makes the behavior policy correlated with outcomes, so directly using logged data for policy evaluation or optimization produces bias, requiring auxiliary structures such as instrumental variables, negative controls or proxy variables to achieve causal identification.

Offline reinforcement learning traditionally assumes that the data come from the target domain and that all confounders are observable, but in real-world scenarios these two assumptions are often broken.

Traditional offline RL methods mitigate distribution shift by imposing divergence constraints on the action distribution (such as KL divergence, Wasserstein distance, MMD, etc.).

The core difficulty of offline reinforcement learning is distribution shift—the learned policy easily deviates from the behavior policy, thereby encountering unseen state-action pairs in the real environment.

Extrapolation error refers to the phenomenon in offline RL where, when the learned policy selects actions that are absent or rare in the dataset, subsequent states enter uncovered regions, causing severe errors in value function estimation that accumulate along trajectory rollout.

Offline reinforcement learning refers to learning a policy solely from a fixed dataset without interacting with the environment, and often requires handling distribution shift.

In action discretization, the dose binning boundaries for vasopressors and fluids directly affect the results of policy learning, and the current binning approach is more heuristic.

ehr2rl is a library written in Python for converting MIMIC-IV electronic medical record data into offline reinforcement learning datasets.

In reward design, mortality is a sparse and delayed reward, whereas SOFA change provides a denser signal but may be inconsistent with long-term outcomes.

Due to the permanent death mechanic, Nuzlocke players deliberately avoid high-risk states, making the conservative strategies prevalent in the dataset naturally aligned with Offline RL optimization objectives.

Offline reinforcement learning penalizes agents for deviating from the dataset distribution by introducing constraint algorithms such as CQL and IQL, but at the cost of limited exploration capability.

Action sequences generated by human players completing Nuzlocke challenges can serve as expert demonstration data for imitation learning/behavioral cloning and offline reinforcement learning.

Offline Reinforcement Learning (Offline RL) uses existing historical data for training, avoiding the risks of online trial-and-error.