PEFT

LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).

Hugging Face has released a series of supporting open-source libraries including transformers, datasets, and diffusers.

During cross-domain fine-tuning, an excessively large learning rate will destroy pre-trained weights, while an excessively small one will result in almost no learning, requiring a careful trade-off.

The core idea of PEFT is to not update all parameters, but instead introduce a small number of trainable adapter parameters (such as LoRA's low-rank matrix decomposition), reducing VRAM and computation requirements while retaining performance close to full fine-tuning.

LoRA reduces the number of trainable parameters by injecting low-rank decomposition matrices into the model's weight matrices, making it possible to fine-tune models with billions of parameters on consumer-grade GPUs.

LoRA (Low-Rank Adaptation) can reduce the number of trainable parameters by more than 99%.

HuggingFace's core products include the Transformers library, the Hub model version management system, and the Inference API.

LoRA将对权重矩阵的全量更新分解为两个低秩矩阵的乘积，训练时只更新这两个小矩阵，原始权重保持冻结

Parameter-efficient fine-tuning (PEFT) methods such as LoRA have significantly lowered the compute barrier for fine-tuning, enabling individual developers to complete fine-tuning on consumer-grade GPUs.

Parameter-efficient fine-tuning (PEFT) methods such as LoRA and Adapter can compress the number of trainable parameters to less than 1% of the total parameters.

PEFT库支持LoRA和QLoRA等参数高效微调方法，可将需要更新的参数量从数十亿降低到数百万级别

Hugging Face Hub is deeply integrated with mainstream open-source libraries such as Transformers, Diffusers, and PEFT, allowing users to load any model with just a single line of code.

The essential theoretical foundations for engineering deployment include the Transformer architecture (understanding the overall process is sufficient) and fine-tuning algorithms (lightweight methods such as LoRA and PEFT).

Hugging Face has become the de facto standard platform in the NLP field, providing a unified interface for using tens of thousands of pre-trained models.

A common combination in practice is: use PEFT to define the LoRA configuration and Unsloth to accelerate training.

Unsloth and approaches such as PEFT and LoRA are complementary rather than substitutes; it is typically used together with parameter-efficient fine-tuning methods, optimizing at the low-level operator layer.

In PEFT, different adapters are mounted on the same layer under namespaces (such as 'default' and 'teacher'), and one of them can be activated via set_adapter().

LoRA/PEFT low-rank updates adjust only a few directions of the weight matrices, which may happen to amplify latent coupling directions that already exist in pretraining, and anomaly detection based on full-parameter changes struggles to capture this targeted amplification.

Parameter-efficient fine-tuning (PEFT) is a class of techniques that adapt to downstream tasks by training only a very small number of newly added or selected parameters, with representative methods including LoRA, prefix tuning, adapters, and so on.

Any study comparing fine-tuning schemes with different numbers of trainable parameters should be wary of the shared learning rate trap; otherwise, conclusions that fewer parameters yield better results may need to be revisited.