Prefix Tuning
A parameter-efficient fine-tuning method that does not alter the model architecture. Instead, it learns a trainable continuous prefix of key-value pairs that is injected into the attention computation of each Transformer layer, adapting to downstream tasks through conditioning in latent space.
Timeline (last 90 days)
Prefix Tuning learns a trainable virtual prefix that serves as additional key-value pairs in attention computation; Prompt Tuning is its simplified variant, learning only soft prompt vectors at the input side.
Prefix Tuning injects trainable prefix key-value pairs into the attention computation of each Transformer layer, whereas Prompt Tuning merely prepends continuous embedding vectors to the input embedding sequence.
Method selection recommendations: prioritize LoRA for low latency, easy deployment, and a mature ecosystem; consider adapters when multi-task isolation and modular mounting are needed; if the parameter budget is extremely tight and tasks are relatively simple, try prefix or prompt tuning.
All Facts (3)
Prefix Tuning learns a trainable virtual prefix that serves as additional key-value pairs in attention computation; Prompt Tuning is its simplified variant, learning only soft prompt vectors at the input side.
50%UnverifiedPrefix Tuning injects trainable prefix key-value pairs into the attention computation of each Transformer layer, whereas Prompt Tuning merely prepends continuous embedding vectors to the input embedding sequence.
50%UnverifiedMethod selection recommendations: prioritize LoRA for low latency, easy deployment, and a mature ecosystem; consider adapters when multi-task isolation and modular mounting are needed; if the parameter budget is extremely tight and tasks are relatively simple, try prefix or prompt tuning.
50%