LoRA
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique for large models. Its core principle is to introduce low-rank decomposition matrices alongside the weight matrices of a pre-trained model, training only these additional parameters while keeping the original model weights frozen. This achieves task-specific adaptation at reduced computational and storage costs. LoRA is widely applied in the customized fine-tuning of large language models and image generation models.
Core Facts
LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).
LoRA achieves fine-tuning by inserting low-rank decomposition matrices alongside the model's weight matrices, typically requiring training of only 0.1%–1% of the original parameter count.
LoRA (Low-Rank Adaptation) was proposed by Microsoft in 2021, with the core idea of freezing the weights of the pretrained model and only training injected low-rank decomposition matrices.
Timeline (last 90 days)
When a text-to-image model generates selfies, character consistency is usually constrained by reference images or character LoRA.
In Unsloth v0.1.913-beta, compatible LoRA and QLoRA tasks on MoE models can be trained together, for example Qwen3.6-35B-A3B, with each task having its own adapter.
A study of LoRA-adapted models reveals that a surprise-driven learner absorbs almost all carefully constructed false claims, with its contamination rate comparable to random selection under the same budget.
Across multiple network architectures and datasets, SAFT leads LoRA-based methods in both clean accuracy and robust accuracy, while using fewer trainable parameters.
Sparse fine-tuning is a class of methods that select a subset of parameters via masks for updating, with representative work including diff pruning, Frank-Wolfe-style sparse fine-tuning, and Lottery Ticket-based subnetwork fine-tuning; compared with structured low-rank methods such as LoRA, it can compress the parameter count to a lower ratio, does not change the model structure, and requires no additional adapter merging at inference.
Parameter-efficient adversarial fine-tuning methods based on LoRA still lag clearly behind full-parameter adversarial training in robustness.
For a d×d weight matrix, LoRA decomposes it into two low-rank matrices A (d×r) and B (r×d), reducing trainable parameters from d² to 2dr, where the rank r is typically 8, 16 or 64.
Small and medium-sized Qwen models can be fine-tuned on a 16GB GPU, achieved with parameter-efficient fine-tuning techniques (such as LoRA) and quantization.
Network rank refers to the intermediate dimension of the low-rank decomposition matrices and determines the feature capacity LoRA can express; too small a rank limits the ability to learn complex concepts, while too large a rank increases the risk of overfitting and the adapter size.
The use of regularization images is disputed in the community: supporters believe they can alleviate overfitting and preserve the base model's general capabilities; opponents believe they add process complexity and their effect varies by task.
40 more timeline events
All Facts (20)
LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).
90%VerifiedLoRA achieves fine-tuning by inserting low-rank decomposition matrices alongside the model's weight matrices, typically requiring training of only 0.1%–1% of the original parameter count.
90%VerifiedLoRA (Low-Rank Adaptation) was proposed by Microsoft in 2021, with the core idea of freezing the weights of the pretrained model and only training injected low-rank decomposition matrices.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedUnsloth is an open-source tooling team focused on large model quantization and fine-tuning acceleration, providing GGUF quantized versions for mainstream open-source models.
85%VerifiedQLoRA was proposed by a research team at the University of Washington in 2023, quantizing base model weights to 4-bit (NF4 format)
85%VerifiedFine-tuning refers to performing secondary training on a foundation model that has completed general-text pre-training, using domain-specific labeled data to adjust the model's parameters
80%VerifiedLoRA decomposes the weight update matrix ΔW into the product of two small matrices A and B whose ranks are much lower than the original matrix, reducing parameter count to as little as one ten-thousandth of the original or even less.
80%VerifiedLoRA is a lightweight model fine-tuning technology that can train a dedicated character model with only 20–50 reference images, requiring far less compute than full fine-tuning.
80%VerifiedLoRA and QLoRA are the currently mainstream parameter-efficient fine-tuning methods, allowing fine-tuning to be completed on consumer-grade GPUs.
80%VerifiedLoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method proposed by Microsoft Research in 2022.
80%VerifiedLoRA (Low-Rank Adaptation) was originally proposed by Microsoft Research for efficient adaptation of large language models.
80%VerifiedLoRA (Low-Rank Adaptation) files are typically only tens of MB in size, far smaller than the several GB size of complete models.
80%Verified"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).
80%VerifiedNeural Tangent Kernel (NTK) theory states that when the width of a neural network approaches infinity, the network's behavior during gradient descent training is equivalent to kernel regression with the NTK as the kernel, and the network does not perform feature learning.
75%VerifiedThe principle of LoRA is to decompose the weight change ΔW during fine-tuning into the product of two low-rank matrices (ΔW = A×B).
75%VerifiedDuring LoRA training, the pre-trained model weights W₀ are frozen and only the two matrices B and A are trained, with the forward propagation formula H = W₀X + (α/r)BAX.
75%VerifiedThe three main strategies for mitigating catastrophic forgetting are regularization-based methods (such as EWC), replay-based methods (such as Experience Replay), and architecture-based methods (such as Progressive Neural Networks).
75%VerifiedThe core idea of PEFT is to not update all parameters, but instead introduce a small number of trainable adapter parameters (such as LoRA's low-rank matrix decomposition), reducing VRAM and computation requirements while retaining performance close to full fine-tuning.
75%VerifiedLoRA reduces the number of trainable parameters by injecting low-rank decomposition matrices into the model's weight matrices, making it possible to fine-tune models with billions of parameters on consumer-grade GPUs.
75%