Continued Pretraining
A training approach in which a general base model is further trained with self-supervised objectives on domain corpora to inject domain knowledge, without alignment to specific instructions.
Timeline (last 90 days)
Just one QLoRA continual pre-training (CPT) run is enough to install a coupled well-being disposition into a model.
The researchers set up a "constructed neutral" control corpus that showed no improvement, indicating the performance gain comes from the corpus content itself rather than the CPT operation.
For small corpora and knowledge-intensive domains (such as healthcare and finance), continued pretraining is still effective and cost-controllable, rather than necessarily requiring direct full-parameter large-scale retraining.
Updating parametric memory requires fine-tuning or continual pre-training, both of which require additional training processes and computational resources.
All Facts (5)
Just one QLoRA continual pre-training (CPT) run is enough to install a coupled well-being disposition into a model.
50%UnverifiedThe researchers set up a "constructed neutral" control corpus that showed no improvement, indicating the performance gain comes from the corpus content itself rather than the CPT operation.
50%UnverifiedFor small corpora and knowledge-intensive domains (such as healthcare and finance), continued pretraining is still effective and cost-controllable, rather than necessarily requiring direct full-parameter large-scale retraining.
50%UnverifiedUpdating parametric memory requires fine-tuning or continual pre-training, both of which require additional training processes and computational resources.
50%Unverified继续预训练（CPT）是在通用基座模型上用领域语料进行额外的无监督训练，让模型吸收领域知识
50%