Pretrained Model
A machine learning model pretrained on large-scale general data that can be transferred to downstream tasks. It is the foundational paradigm of modern NLP and AI applications.
Timeline (last 90 days)
This study provides a rigorous theoretical foundation for transfer learning, demonstrating its scaling advantage, meaning that the reuse of pretrained models has a theoretical optimality guarantee.
Experimental results show that pretraining does produce structure beyond a random kernel, but far from that of a linear world model, and the representation geometry learned by the models remains highly curved.
Facts in pretrained models can be divided into two types, active facts and inert facts; active facts can support generalization beyond the memorized range after fine-tuning, while inert facts can be recalled but cannot produce generalization on the same task.
In an image classification experiment, the pretrained model returned multiple candidate labels with scores: Website at 87.5%, Menu at 8.4%, Screen at 2.2%, Television at 0.7%, and Brass at 0.4%.
The widespread adoption of large pre-trained models has significantly raised baseline performance on many tasks, making purely incremental performance gains increasingly difficult to impress.
LoRA fine-tuning based on pre-trained models is the most realistic technical path for individual developers.
现代AI开发的主流工作流是通过Hugging Face拉取预训练权重，而非从零训练模型
使用开源预训练模型的权重与使用他人的私有训练数据是两件完全不同的事情
All Facts (7)
This study provides a rigorous theoretical foundation for transfer learning, demonstrating its scaling advantage, meaning that the reuse of pretrained models has a theoretical optimality guarantee.
50%UnverifiedExperimental results show that pretraining does produce structure beyond a random kernel, but far from that of a linear world model, and the representation geometry learned by the models remains highly curved.
50%UnverifiedFacts in pretrained models can be divided into two types, active facts and inert facts; active facts can support generalization beyond the memorized range after fine-tuning, while inert facts can be recalled but cannot produce generalization on the same task.
50%UnverifiedThe widespread adoption of large pre-trained models has significantly raised baseline performance on many tasks, making purely incremental performance gains increasingly difficult to impress.
50%UnverifiedLoRA fine-tuning based on pre-trained models is the most realistic technical path for individual developers.
50%Unverified现代AI开发的主流工作流是通过Hugging Face拉取预训练权重，而非从零训练模型
50%Unverified使用开源预训练模型的权重与使用他人的私有训练数据是两件完全不同的事情
50%