Permanent and Transient Representations: Solving the Stability-Plasticity Dilemma in Continual Reinforcement Learning

Continual Reinforcement Learning (CRL) is fundamentally defined by the trade-off between stability and plasticity: an agent must rapidly adapt to new environments while not forgetting old knowledge. Traditional methods often achieve one at the expense of the other. A new study, drawing on complementary learning systems, proposes using parametric and non-parametric representations to serve long-term generalization and short-term adaptation respectively, and validates its effectiveness on image-based tasks and Craftax-Classic.

Stability and Plasticity: The Fundamental Dilemma of Continual Reinforcement Learning

In continual reinforcement learning, agents face a dilemma: if they update models too heavily on new data, performance on old tasks rapidly degrades (catastrophic forgetting); if they cling to existing knowledge, they struggle to adapt to new situations. This stability-plasticity trade-off has long been a bottleneck preventing agents from learning continually in open worlds.

The Complementary Learning Systems (CLS) theory offers inspiration for this challenge: in biological brains, the hippocampus rapidly encodes specific experiences, while the neocortex forms generalized knowledge through slow statistical learning. Anand and Precup (2023) introduced this idea to reinforcement learning, using a permanent value function and a transient value function to correspond to long-term learning and rapid adaptation, respectively. However, in their implementation, both systems used the same feature representation, which somewhat weakened the complementary advantage.

Catastrophic Forgetting refers to the phenomenon where, when a neural network learns a new task, parameter updates overwrite the weights that previous tasks relied on, causing a sharp drop in performance on old tasks. This phenomenon is particularly severe in reinforcement learning because the data distribution constantly drifts as the policy changes, and environment interaction samples are highly correlated. Common mitigation methods include experience replay (storing and replaying old data), regularization (constraining changes to important parameters), and dynamically expanding network capacity. Continual reinforcement learning is proposed precisely in this context, requiring an agent to sequentially learn multiple tasks on a single model without forgetting.

Parametric vs. Non-Parametric: Giving Each System Its Own Role

The key insight of the new study is that long-term learning should rely more on parametric representations to achieve broad generalization, while transient learning is better suited to non-parametric representations to quickly capture situation-specific details. Parametric models (such as neural networks) gradually compress abstract features during slow learning, but updates are costly; non-parametric methods (such as instance-based retrieval) can adapt quickly to new situations without training, but struggle to generalize.

The authors explored this design both theoretically and experimentally. Theoretically, they analyzed the differences between the two representations in terms of the bias-variance trade-off; experimentally, they proposed a novel non-parametric approximation method for estimating the transient value function in complex tasks. This method requires no gradient updates for the transient system; instead, it responds quickly by retrieving similar historical experiences, thereby enabling online learning.

Non-parametric methods in continual learning typically refer to models that can make predictions without relying on gradient training, with instance-based retrieval and nearest-neighbor methods as typical representatives. Their advantage is that new samples can immediately participate in inference simply by being stored in an experience pool, providing fast adaptation without repeated training; the disadvantage is that storage and retrieval costs grow linearly with the number of samples, and they are sensitive to similarity metrics in high-dimensional inputs. In contrast, parametric representations (such as neural network weights) can compress knowledge into fixed-dimensional vectors after training, offering strong generalization and low inference cost, but with high update costs and a tendency to overwrite old knowledge. The two are exactly complementary in the bias-variance trade-off: parametric models lean toward low variance and high bias, while non-parametric methods lean toward low bias and high variance.

Experimental Validation: Advantages on Image Tasks and Craftax-Classic

On image-input tasks and in the Craftax-Classic environment, the method demonstrated performance superior to competing baselines. Craftax-Classic is a complex pixel-based survival game environment that places high demands on an agent's perception and decision-making abilities. Experimental results show that pairing the permanent value function with parametric representations and the transient value function with non-parametric representations can significantly improve online learning efficiency and alleviate the stability-plasticity conflict.

This result cross-validates with several independent analyses: according to discussions of complementary learning systems by Bilibili creators and YouTube channels, system-level decomposition—rather than unified updates of a single model—is key to improving the robustness of continual learning. At the same time, some researchers point out that the computational overhead of non-parametric methods grows with the experience pool, requiring a trade-off in retrieval efficiency for practical deployment. Nevertheless, the study provides a scalable architectural approach for continual reinforcement learning.

Craftax-Classic is a JAX-based Crafter-style survival game environment in which agents learn skills such as gathering resources, maintaining health, and exploring the map solely from pixel observations. Compared with benchmarks like Atari, it emphasizes long-horizon planning and exploration under sparse rewards, and its high environment stochasticity poses a stricter test of rapid adaptation in continual learning. This environment is often used to evaluate sample efficiency and generalization ability, making it a suitable testbed for examining whether the permanent/transient value function decomposition is effective.

Implications for Continual Learning Architectures

The value of this work lies not only in performance improvements but also in clarifying the necessity of representation separation. Many previous continual learning methods attempted to use the same network for both stability and plasticity, whereas this study demonstrates that decoupling long-term generalization from short-term adaptation—entrusting them to parametric and non-parametric systems respectively—can more closely approximate the complementary learning mechanisms of biological brains.

Of course, the method still has limitations: the experience storage and retrieval costs of the non-parametric transient system grow with the number of tasks, and long-term scalability remains to be further verified. In addition, it has so far only been tested on image tasks and Craftax-Classic; extending it to more complex, high-dimensional continuous control tasks is a future direction. Nonetheless, this study provides a new principle for the architectural design of continual reinforcement learning: the form of representation should match the time scale of learning.

One-line Summary

Using parametric and non-parametric representations to serve long-term generalization and short-term adaptation respectively, alleviating the stability-plasticity dilemma in continual reinforcement learning.

Paragraph Summary

The article centers on the fundamental stability-plasticity dilemma in continual reinforcement learning. Drawing on complementary learning systems theory, it assigns long-term learning and rapid adaptation to a permanent value function and a transient value function, respectively. The key improvement of the new study lies in representation separation: the permanent system uses a parametric neural network for broad generalization, while the transient system uses non-parametric retrieval for rapid responses without gradient updates, with theoretical analysis provided from the perspective of the bias-variance trade-off. On image tasks and in the Craftax-Classic environment, the method outperforms competing baselines and improves online learning efficiency. The article also notes that the storage and retrieval overhead of the non-parametric experience pool grows with the number of tasks, and long-term scalability and extension to continuous control tasks remain open problems.