Piano Transcription Model Fine-Tuned for Guitar Suffers Catastrophic Forgetting—Where's the Problem?

A developer reached out on Reddit: he tried to fine-tune a piano transcription model for guitar, but ran into severe catastrophic forgetting—the model made slight progress on guitar data, yet its original piano transcription ability collapsed dramatically. This is almost the most classic failure scenario in cross-instrument transfer learning.

Why Catastrophic Forgetting Targets Transcription Tasks in Particular

The output of a transcription model is dense frame-by-frame or note-wise predictions. Piano and guitar differ enormously in range, timbre, and playing technique, so the feature distributions the model learns are inherently inconsistent. When you directly fine-tune the entire network with a small amount of guitar data, gradient updates quickly overwrite the weights originally learned for piano, causing the old task's performance to plummet.

What's worse, transcription tasks are extremely sensitive to temporal alignment and pitch accuracy. Even if the weights are only slightly perturbed, the output may degrade from "basically accurate" to "completely unusable." This explains why the forgetting the developer experienced was more severe than in typical classification tasks.

Catastrophic forgetting is a typical failure mode of neural networks when learning multiple tasks sequentially: when the model continues training on a new task, gradient updates overwrite the weights learned for the old task, causing a sharp decline in old-task performance. This phenomenon was systematically described by McCloskey and Cohen in 1989 and later became a core problem in the field of continual learning. Its root cause lies in the parameter-sharing mechanism of neural networks—the same set of weights simultaneously encodes solutions for multiple tasks, and standard fine-tuning has no explicit constraint to protect the key parameters of the old task. Mitigation approaches fall roughly into three categories: regularization methods (such as EWC, which penalizes changes to parameters important for the old task), replay methods (mixing in old data or generating pseudo-samples), and structural methods (isolating some parameters or using adapters). Understanding this classification helps determine which technical route each of the fixes mentioned in this article belongs to.

Common Mistakes and Corresponding Fixes

Full Fine-Tuning + Small Dataset Is the Most Dangerous Combination

If the amount of guitar data is limited, directly unfreezing all parameters for fine-tuning will almost inevitably trigger forgetting. A safer approach is to first freeze the backbone network and train only the final output head or adapter layers, letting the model learn the guitar mapping while preserving piano features.

No Differentiation in Learning Rates

Using the same large learning rate as during pretraining in the fine-tuning stage amplifies weight drift. It is recommended to use a smaller learning rate for pretrained layers and a normal learning rate for newly added layers, forming a layered learning rate strategy.

Lack of Replay or Regularization Constraints for the Old Task

Mixing in some piano data during fine-tuning, or introducing regularization terms such as Elastic Weight Consolidation (EWC), can significantly alleviate forgetting. Even retaining only a small number of old-task samples can provide "anchors" for the model to prevent parameters from drifting.

Elastic Weight Consolidation (EWC) was proposed by Kirkpatrick et al. in 2017 and is one of the most representative regularization methods in continual learning. Its core idea is: first use the Fisher Information Matrix to estimate the importance of each parameter to the old task, then apply a quadratic penalty to important parameters during subsequent fine-tuning so they are not easily changed substantially, thereby balancing "learning new" and "preserving old." EWC requires no storage of old-task data, making it suitable for scenarios with data privacy or storage constraints; however, its approximation assumptions are strong (such as a local quadratic approximation of the loss surface), and its effectiveness is limited when tasks differ greatly. For output-dense tasks like transcription, EWC can be combined with a small amount of data replay to form a dual constraint.

Cross-Instrument Transfer: Is There a Less Troublesome Path?

If the goal is merely to make the model "able to recognize guitar" rather than completely replace piano ability, multi-task learning can be considered: have the model output transcription results for both piano and guitar simultaneously, share underlying features, and let the two tasks constrain each other. Another approach is to adopt an adapter structure, inserting a small number of trainable modules into a frozen backbone, thereby preserving the original ability while acquiring a new skill.

According to common experience in community discussions, the larger the parameter count of a pretrained model, the more likely it is to "remember new things and forget old things" during fine-tuning. Therefore, in small-data scenarios, parameter-efficient fine-tuning (PEFT) solutions should be prioritized over brute-force full fine-tuning.

Parameter-Efficient Fine-Tuning (PEFT) is an umbrella term for methods that train only a very small number of newly added parameters while freezing the pretrained backbone. Common forms include Adapter, Low-Rank Adaptation (LoRA), Prefix Tuning, etc. Its advantages are: the number of trainable parameters is usually less than 1% of the total, greatly reducing the risk of overfitting and forgetting in small-data scenarios; meanwhile, each task can be saved as an independent lightweight module, facilitating multi-task switching and deployment. The adapter structure mentioned in the article is an early representative of PEFT—inserting small feed-forward modules between Transformer layers and updating only these modules during training. For cross-instrument transcription, PEFT can keep piano ability in the frozen backbone while concentrating guitar ability in a small number of trainable parameters, structurally isolating old and new tasks.

A Checklist for Developers Facing Similar Problems

• Confirm whether the amount of fine-tuning data matches the model scale; if data is scarce, don't do full fine-tuning;
• Check whether the backbone is frozen and whether learning rates are differentiated;
• Retain a certain proportion of old-task data in the training set for replay;
• Monitor old-task validation set metrics and adjust strategies promptly once they decline significantly;
• Consider multi-task or adapter solutions to structurally isolate old and new tasks.

Catastrophic forgetting is not mysticism but the inevitable result of a mismatch between fine-tuning strategy and data distribution. Treat "preserving the old" and "learning the new" as two objectives that need to be optimized simultaneously, and the problem often resolves itself.