PNSN Experimental Architecture Sparks Discussion: Can Modular Retrieval Break Through Catastrophic Forgetting?

A developer has publicly released an experimental AI architecture called PNSN (Persistent Neural State Network), attempting to solve a long-standing problem in the large-model space—catastrophic forgetting—through parameter modularization and a trainable retrieval mechanism. Although the initial experimental results are modest, the approach has prompted the technical community to re-examine modular network design.

Architecture Core: Parameter Collections and Trainable Retrieval

The basic design of PNSN is not complicated: learned parameters are organized into independent collections, each of which can be viewed as a subnetwork specializing in particular knowledge or patterns. When the model receives a query, it selects the most relevant collection to participate in computation through indexing and a trainable retrieval mechanism (IRM).

This design stands in sharp contrast to today's mainstream large monolithic models. In a traditional Transformer, all parameters are activated during every forward pass, so growth in model size means a linear increase in computational cost. PNSN attempts to decouple "knowledge storage" from "routing selection," so that the model only invokes the necessary parameter modules.

In the Reddit post, the developer openly stated that they designed the architecture and led the research and implementation, but that an AI assistant helped generate a considerable proportion of the code. This candor is commendable—in an era where AI-assisted programming is increasingly common, being transparent about the boundaries of tool use is more conducive to technical discussion than pretending to have handwritten every line of code.

Identity Baseline in retrieval experiments refers to a routing strategy that performs no learning at all: regardless of what the input query is, the IRM directly returns the query's own original representation (or a fixed mapping), without any parameterized selection or transformation. It serves as a lower-bound reference akin to "random guessing" or "no learning," used to measure whether the trained retrieval mechanism has truly learned meaningful collection-selection capability. If the trained IRM is only slightly above the identity baseline, it suggests that its routing decisions may mainly come from surface features of the input rather than genuine semantic matching; if it is significantly above the baseline, it indicates that trainable retrieval has indeed captured the correspondence between queries and collections. With an extremely small sample size, the statistical significance of the baseline comparison is limited, so the developer emphasized that the results are only a preliminary signal.

Retrieval Experiment: A Preliminary Signal of 4/7 versus 3/7

PNSN's core mechanism, IRM, underwent a small evaluation: using seven held-out query formulations, the trained IRM achieved 4/7 correct on top-1 results, while the identity baseline was 3/7. On the identity-related subset, the contrast was more pronounced—3/5 versus 1/5.

The developer remained cautious about this: "These results are preliminary. The experiments so far use only three collections and five training samples, so they cannot establish robust generalization, nor can they prove that PNSN prevents catastrophic forgetting."

This self-limitation deserves to be taken seriously. Seven queries and three collections are far from sufficient to reach a statistically significant conclusion. However, the trainability of the retrieval mechanism itself—rather than fixed-rule routing—is a meaningful direction to explore. If IRM can learn meaningful collection-selection strategies at a larger scale, then the feasibility of modular architectures will receive substantive support.

Unresolved Challenges: Heterogeneous Collections and Catastrophic Forgetting

The developer specifically mentioned an issue that has not yet been tested: can collections with different network architectures or parameter shapes coexist in the same system and be retrieved?

This question touches the core contradiction of modular AI. If all collections must share the same topology, then the flexibility of modularization is severely limited. Conversely, if heterogeneous collections are allowed to coexist, the retrieval mechanism must be compatible across architectures, which is far more complex than selecting within a homogeneous space.

As for catastrophic forgetting, PNSN's hypothesis is: independent parameter collections mean that updating one collection will not overwrite the knowledge of other collections. But this hypothesis requires rigorous experimental verification—in continual learning scenarios, the retrieval mechanism itself may drift, causing originally correct routing decisions to fail.

Catastrophic Forgetting is a classic defect of artificial neural networks in continual learning: when a model learns multiple tasks in sequence, the gradient updates from a new task can substantially overwrite the weights on which previous tasks depend, causing the model's performance on earlier tasks to drop sharply. This essentially stems from parameter sharing—the same set of weights is reused by all tasks, so updates inevitably cause interference. Common mitigation strategies include Elastic Weight Consolidation (EWC), experience replay, and parameter isolation. PNSN belongs to the parameter isolation route: by distributing knowledge across independent collections and activating them on demand through trainable routing, it can theoretically avoid affecting other collections when updating one collection, thereby mitigating forgetting. However, if the routing mechanism itself drifts during training, old knowledge may become "unfindable," which is precisely the key risk that needs experimental verification.

Community Feedback and Open-Source Invitation

The developer hosts the code in a GitHub repository (pathetictype-bit/Apex) and explicitly states that they hope other developers will help investigate the architecture, reproduce results, find flaws, or challenge the underlying assumptions.

This open attitude has practical value in the AI research community. Independent researchers are constrained by local hardware and find it difficult to conduct large-scale experiments, but community collaboration can partially make up for this shortcoming. The developer also admitted: "My local hardware limits how far I can test certain experiments."

From the perspective of technical discussion, PNSN currently looks more like a prototype hypothesis worth taking seriously than a usable system. Its value lies not in the answers it provides, but in the questions it raises: when we have a trainable routing mechanism, can parameter modularization really become a viable path toward continual learning? This question deserves more experiments to answer.

One-line Summary

A developer has publicly released an experimental architecture, PNSN, using parameter modularization and trainable retrieval to combat catastrophic forgetting; the preliminary results are modest but have prompted the community to re-examine modular networks.

Paragraph Summary

The developer has publicly released an experimental AI architecture called PNSN (Persistent Neural State Network), attempting to solve the catastrophic forgetting problem of large models through parameter modularization and a trainable retrieval mechanism. PNSN organizes learned parameters into independent collections, each specializing in particular knowledge, and at query time a trainable retrieval mechanism, IRM, selects the most relevant collection to participate in computation, thereby decoupling knowledge storage from routing selection. The preliminary retrieval experiment achieved a top-1 accuracy of 4/7 on seven held-out queries, versus 3/7 for the identity baseline, and 3/5 versus 1/5 on the identity-related subset, but it used only 3 collections and 5 training samples, so statistical power is limited. The developer candidly acknowledged that an AI assistant generated a large amount of the code and, due to local hardware constraints, invited the community to reproduce the work and challenge the assumptions. Coexistence of heterogeneous collections and rigorous verification of catastrophic forgetting remain unresolved challenges.