Prompt Tuning
A parameter-efficient fine-tuning method that prepends trainable continuous embedding vectors, known as a soft prompt, to the input embedding sequence. Only these vectors are optimized while the model parameters remain unchanged, enabling lightweight task adaptation.
Timeline (last 90 days)
Prefix Tuning learns a trainable virtual prefix that serves as additional key-value pairs in attention computation; Prompt Tuning is its simplified variant, learning only soft prompt vectors at the input side.
Prefix Tuning injects trainable prefix key-value pairs into the attention computation of each Transformer layer, whereas Prompt Tuning merely prepends continuous embedding vectors to the input embedding sequence.
The continuous embedding vectors learned by prompt tuning do not correspond to any real token in the vocabulary, so they lose readability and cannot be directly translated into human-understandable natural language instructions.
Method selection recommendations: prioritize LoRA for low latency, easy deployment, and a mature ecosystem; consider adapters when multi-task isolation and modular mounting are needed; if the parameter budget is extremely tight and tasks are relatively simple, try prefix or prompt tuning.
Prompt Tuning was proposed by Lester et al. in 2021, prepending a set of learnable soft prompt vectors before the input embeddings.
All Facts (5)
Prefix Tuning learns a trainable virtual prefix that serves as additional key-value pairs in attention computation; Prompt Tuning is its simplified variant, learning only soft prompt vectors at the input side.
50%UnverifiedPrefix Tuning injects trainable prefix key-value pairs into the attention computation of each Transformer layer, whereas Prompt Tuning merely prepends continuous embedding vectors to the input embedding sequence.
50%UnverifiedThe continuous embedding vectors learned by prompt tuning do not correspond to any real token in the vocabulary, so they lose readability and cannot be directly translated into human-understandable natural language instructions.
50%UnverifiedMethod selection recommendations: prioritize LoRA for low latency, easy deployment, and a mature ecosystem; consider adapters when multi-task isolation and modular mounting are needed; if the parameter budget is extremely tight and tasks are relatively simple, try prefix or prompt tuning.
50%UnverifiedPrompt Tuning was proposed by Lester et al. in 2021, prepending a set of learnable soft prompt vectors before the input embeddings.
50%