Continual Learning of New Facts Without Forgetting: A Four-Night Showdown Between LoRA and a Custom Method

Article Content

Can large models absorb new knowledge every day like humans without forgetting old skills? A four-day experiment delivered a stark comparison: fine-tuning with LoRA to learn the day's news allowed the model to still answer 96% of new questions on the second day, but after four days performance plummeted to 26.7%, and math ability nearly vanished. Another undisclosed method kept first-day knowledge stable above 90%, with math and general capabilities essentially intact. This test, initiated by Reddit user fintanlabs, has had its dataset published on Hugging Face, sparking discussion about continual learning and catastrophic forgetting.

Experimental Design: Four Days of News, Nightly Updates, Repeated Interrogation

The experiment's core is to simulate the scenario of "writing each day's news facts into the model weights every night." The researchers chose the Apertus-8B model and fine-tuned it with real news data for four consecutive days. After each day's training, the model was re-questioned on news from all previous days to test the degree of forgetting. The pass bar was determined before question generation, and a strict scorer was used to avoid post-hoc adjustment of standards. All questions and answers have been made public, ensuring reproducibility.

According to the original Reddit post, the test focused on "next-day questions"—i.e., asking about the previous day's news the day after training. LoRA achieved 93% to 96% accuracy on the first day, with no context needed in the prompt. However, as nights passed, old knowledge rapidly disintegrated.

Apertus-8B is an open-source large language model released by the Swiss AI Initiative in 2025, with 8 billion parameters, supporting multilingual and long contexts. The model was trained under a fully transparent data governance framework, with weights and training details publicly available, so it is often chosen as a baseline model in academic research. In this experiment, choosing Apertus-8B rather than a commercial closed-source model facilitates reproducibility and attribution, but the 8B scale also means its knowledge capacity and anti-forgetting ability may be weaker than larger models; this factor should be considered when interpreting the results.

LoRA's Collapse: New Knowledge Learned, Old Abilities Lost

LoRA (Low-Rank Adaptation), as a lightweight fine-tuning method, is often used to inject new knowledge. But experimental data show it has serious flaws in continual learning scenarios.

First-day news Q&A accuracy started at 96%, and after the second, third, and fourth nights of training, dropped to 52%, 32%, and 26.7%, respectively. More worrying, the model's mathematical reasoning ability (GSM8K) plummeted from 69.6% to 4.4%. This means the model, while absorbing new facts, almost completely lost its ability to solve math problems. The original post bluntly stated: "It can't do math anymore."

This catastrophic forgetting is not unique to LoRA, but it is particularly pronounced under this experiment's setup. Each fine-tuning overwrote the previous weight adjustments, causing old knowledge to be rapidly washed away.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Its core idea is to freeze most of the pretrained model's weights and inject two low-rank matrices (A and B) into each layer, adjusting model behavior by training these small matrices. Since the rank is much smaller than the original weight dimensions, LoRA's trainable parameters typically account for only 0.1%-1% of the full model, greatly reducing computation and storage overhead. However, this low-rank constraint also means model updates are restricted to a low-dimensional subspace. In continual learning scenarios, each new task training overwrites the previous low-rank matrices, causing old knowledge to be rapidly washed away—this is precisely the structural cause of LoRA's catastrophic forgetting in the experiment.

Custom Method: Knowledge Remains Stable, General Abilities Intact

Compared with LoRA, the researchers also tested an undisclosed method. After four nights of training, this method kept first-day news Q&A accuracy at 90.7%. Math ability GSM8K was 69.2%, essentially on par with the baseline of 69.6%; MMLU (Massive Multitask Language Understanding) score was 58.4%, slightly above the baseline of 57.4%. This indicates the method effectively avoids catastrophic forgetting while injecting new knowledge.

However, the method is not perfect. The original post noted that on second- and third-day news, accuracy dropped to about 70%, especially when the same story appeared again on subsequent days. This may be because the model experienced interference when processing repeated information. The researchers also emphasized that the method has not been made public and therefore cannot be independently verified.

GSM8K (Grade School Math 8K) is a benchmark dataset for evaluating large models' mathematical reasoning, containing about 8,500 grade-school math word problems requiring multi-step arithmetic reasoning. MMLU (Massive Multitask Language Understanding) covers multiple-choice questions across 57 subjects, measuring models' general knowledge and reasoning. The two are often used together: GSM8K reflects chain-of-thought reasoning, MMLU reflects breadth of knowledge. In the experiment, GSM8K plummeted from 69.6% to 4.4%, meaning the model almost entirely lost multi-step calculation ability, while MMLU remained relatively stable, indicating factual knowledge suffered less damage.

Multi-Source Perspectives: Community Discussion and Controversies

After the experiment was posted on Reddit's MachineLearning subreddit, it sparked discussion about continual learning methodologies. Some commenters believed LoRA's collapse was not surprising, because full-parameter fine-tuning itself causes catastrophic forgetting, and LoRA's low-rank constraint may exacerbate the problem. Others questioned the reproducibility of the custom method, since the code was not released.

According to the Hugging Face dataset page, all question-answer pairs have been made public, but method details are missing. Some researchers noted that the experiment's pass bar was strict and predefined, increasing the credibility of the results. However, due to the lack of peer review, conclusions should still be treated with caution.

Implications: Continual Learning Still Needs Breakthroughs

This four-day experiment visually demonstrates the dilemma of current large models in continual learning: either forgetting old knowledge quickly or absorbing new knowledge poorly. LoRA's failure reminds us that simple fine-tuning strategies struggle with dynamic knowledge updates. Although the custom method performs better, its undisclosed details and performance drops on the second and third days indicate there is still a long way to go in this field.

For practitioners, public experimental data is a positive signal. In the future, how to balance plasticity and stability will be key to continual learning. Perhaps methods combining memory replay, regularization, or architectural improvements can truly enable models to "learn every night without collapsing."

One-line Summary

A four-day continual learning experiment shows LoRA suffers severe forgetting, while a custom method remains stable but its details are undisclosed.

Paragraph Summary

A four-day experiment initiated by a Reddit user compared LoRA with an undisclosed method in continual learning. LoRA achieved 96% accuracy on first-day news Q&A but plummeted to 26.7% after four days, with math ability GSM8K dropping from 69.6% to 4.4%; the custom method maintained 90.7% accuracy on first-day knowledge, with math and general abilities on par with baseline, but second- and third-day news accuracy dropped to about 70%. The experiment used a predefined pass bar and strict scorer, and Q&A data was published on Hugging Face, but custom method details are missing. The results highlight the tension between plasticity and stability in continual learning, and simple fine-tuning like LoRA struggles to avoid catastrophic forgetting.