Can You Fine-Tune by Deleting Weights? The Deletion Tuning Method Draws Attention

Title

Can You Fine-Tune by Deleting Weights? The Deletion Tuning Method Draws Attention

Meta Description

Deletion Tuning matches full fine-tuning in reinforcement learning, supervised fine-tuning, and on-policy distillation by deleting less than 0.05% of weights, while drastically reducing checkpoint size and mitigating catastrophic forgetting. Geometric analysis reveals why it works.

Article Content

LLM post-training typically means adjusting a large number of parameters, but a new study proposes a completely different approach: deleting only a tiny fraction of weights can achieve the same effect as full fine-tuning. This discussion from Reddit focuses on a method called Deletion Tuning, which zeroes out less than 0.05% of the weights in the gated projection of Transformer blocks, matching full fine-tuning (FFT) performance across three post-training scenarios: reinforcement learning, supervised fine-tuning, and on-policy distillation.

Core Finding: Deleting Is More Effective Than Adjusting

The research team found that pretrained large models do not need large-scale weight modifications to acquire new capabilities. Their proposed Deletion Tuning method learns a binary mask applied to the gated projection of Transformer blocks, zeroing out selected weights. Surprisingly, this minimalist operation matches full fine-tuning performance across all three post-training paradigms:

• Reinforcement learning with verifiable rewards (RLVR)
• Supervised fine-tuning (SFT)
• On-policy distillation

This means that adapting model capabilities may depend more on suppressing certain existing connections than on reshaping a large number of parameters. This finding challenges the conventional wisdom that "fine-tuning must involve substantial weight adjustments."

Gated projection is a key component in the feed-forward network (FFN) or gated linear unit (GLU) structure within Transformer blocks. Taking the common SwiGLU as an example, the FFN contains three projection matrices: a "gate" projection that produces a gating signal passed through an activation function, an "up" projection that provides the features to be gated, and after element-wise multiplication, a "down" projection that maps back to the model dimension. The gated projection determines which intermediate features are amplified or suppressed, directly affecting the model's expressiveness and information routing. Deletion Tuning applies a binary mask to the weights of the gated projection, meaning it precisely intervenes in the feature selection stage rather than randomly deleting parameters. This choice of location may be an important reason why the method can match full fine-tuning with such a minimal amount of deletion.

Two Practical Advantages: Storage and Forgetting

Deletion Tuning brings two direct benefits.

Dramatically Reduced Checkpoint Size

Since only a binary mask is learned, the resulting mask is hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoint. This is significant for model deployment and version management—storage and transfer costs are substantially reduced, making it especially suitable for scenarios that require frequent task switching.

A checkpoint is a snapshot of the model state saved during training, typically containing all trainable parameters. Therefore, a full fine-tuning checkpoint is comparable in size to the base model, often tens of GB. In multi-task deployment, if a complete checkpoint is saved for each task, storage overhead grows linearly with the number of tasks. Deletion Tuning only learns a binary mask with the same shape as the gated projection, where each element requires just 1 bit, so its size can be hundreds to thousands of times smaller than a full checkpoint. At deployment, you only need to save the base model and the mask; at inference, zeroing out the corresponding weights according to the mask restores the fine-tuned effect. This "base + mask" paradigm also simplifies version management: switching tasks only requires replacing the mask file rather than moving an entire model.

Mitigated Forgetting in Continual Learning

A core challenge in continual learning is catastrophic forgetting. In the reinforcement learning setting, Deletion Tuning maintains prior task accuracy near the base model level, while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points. This comparison shows that deleting weights causes far less damage to existing knowledge than traditional fine-tuning.

Catastrophic forgetting refers to the phenomenon where, when a model learns a new task, gradient updates on the new task overwrite or interfere with the weights that prior tasks depend on, causing a sharp decline in old task performance. In continual learning scenarios, where the model must sequentially adapt to multiple tasks, the forgetting problem is particularly pronounced. Traditional full fine-tuning causes the most damage to old knowledge due to large weight changes; although LoRA only updates low-rank adapters, the adapter combined with the original weights can still shift the original representations. Deletion Tuning only zeroes out a small number of weights, essentially performing "subtraction" on the original network, which preserves undeleted weights more completely. As a result, in the reinforcement learning setting, old task accuracy can remain near the base model level. This suggests that the degree of damage to old knowledge during fine-tuning is not always proportional to the magnitude of weight changes—the "location" and "manner" of changes are equally important.

Geometric Analysis: Why Is a Mask Sufficient?

To understand why a mask can substitute for weight updates, the researchers analyzed the learned weight changes under reinforcement learning from a geometric perspective. They used three metrics:

• Stable rank
• Spectrum drift
• Principal-weight overlap

Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to full fine-tuning on all three metrics. This suggests that the effective changes of full fine-tuning may be concentrated on a small number of key weights, and deleting these weights恰好 simulates this concentrated adjustment.

Stable rank is a normalized measure of a matrix's singular values, defined as the ratio of the squared Frobenius norm to the squared spectral norm. Unlike ordinary rank, it reflects the "effective dimensionality" of the singular value distribution: if a few singular values are much larger than the rest, the stable rank will be significantly lower than the matrix dimension, indicating that the matrix's energy is concentrated in a few directions. When analyzing fine-tuning, stable rank can be used to determine whether weight updates are concentrated—the lower the stable rank, the more concentrated the updates are in a few principal directions.

Spectrum drift measures the overall shift of the singular value spectrum of the weight matrix after fine-tuning relative to the pretrained weights, typically quantified by the distance between singular value vectors. Small spectrum drift means that fine-tuning has not substantially reshaped the matrix's spectral structure, and the original information channels are largely preserved.

Principal-weight overlap focuses on the degree of alignment between the dominant singular vectors (i.e., the directions corresponding to the largest singular values) before and after fine-tuning. High overlap indicates that fine-tuning primarily proceeds along pre-existing principal directions rather than opening up entirely new ones. Together, these three metrics characterize the "concentration" and "continuity" of weight changes, which is precisely why Deletion Tuning is closest to full fine-tuning.

Implications for Fine-Tuning Practice

This study offers a new perspective on parameter-efficient fine-tuning. Traditional methods like LoRA adjust weights through low-rank adapters, while Deletion Tuning directly deletes weights—two fundamentally different approaches. However, the former requires introducing additional parameters, while the latter only needs a mask, giving it advantages in storage and forgetting control.

That said, the method is still in the research stage, and its generality and performance across different model scales remain to be further validated. For practitioners focused on continual learning and efficient fine-tuning, this is undoubtedly a direction worth following.

Summary

Deletion Tuning zeroes out less than 0.05% of weights, matching full fine-tuning across multiple post-training tasks while dramatically reducing checkpoint size and catastrophic forgetting. Geometric analysis further supports its effectiveness. This research, discussed on Reddit, opens up a counterintuitive yet promising path for efficient fine-tuning.

One-line Summary

Deletion Tuning matches full fine-tuning across multiple post-training tasks by deleting less than 0.05% of weights.

Paragraph Summary

This article introduces a post-training method called Deletion Tuning: instead of adjusting a large number of parameters, it learns a binary mask that zeroes out less than 0.05% of the weights in the gated projection of Transformer blocks. Across reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation, the method performs comparably to full fine-tuning. Its advantages include checkpoints that are hundreds to thousands of times smaller than full fine-tuning, and significantly mitigated catastrophic forgetting in continual learning—old task accuracy under reinforcement learning stays close to the base model, whereas full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points. Geometric analysis shows that Deletion Tuning is closest to full fine-tuning on stable rank, spectrum drift, and principal-weight overlap, suggesting that effective fine-tuning may concentrate on suppressing a few key weights rather than large-scale reshaping.