Hinton
Geoffrey Hinton is a British-Canadian computer scientist and one of the founding figures in artificial intelligence and machine learning. He has devoted his career to foundational research in neural networks and deep learning, making pioneering contributions to the popularization of the backpropagation algorithm and methods for training deep neural networks. He held positions at the University of Toronto and Google, and is regarded as a central scholar in bringing deep learning into mainstream AI research. He is a recipient of the Turing Award and numerous other major academic honors.
Core Facts
Knowledge Distillation is a technique for transferring knowledge from a large teacher model to a smaller student model, proposed by Geoffrey Hinton et al. in 2015
The backpropagation algorithm was systematized by Rumelhart, Hinton, and others in a paper published in Nature in 1986.
Knowledge Distillation was originally proposed by Hinton et al. in 2015, with the core idea of having a small model learn the soft labels of a large model rather than hard labels.
Timeline (last 90 days)
Model distillation is a model compression and knowledge transfer technique first proposed by Hinton et al. in 2015.
The concept of fast weights can be traced back to the fast weight associative memory proposed by Hinton and von der Malsburg in the 1990s, as well as Schmidhuber's linear Transformer work.
In 1986, Rumelhart, Hinton, and Williams published the paper "Learning representations by back-propagating errors" in the journal Nature.
The paper's signatories include Anthropic co-founder Jack Clark and OpenAI's Chief Scientist.
Hinton proposed the Forward-Forward Algorithm in 2022, replacing one forward pass plus one backward pass with two forward passes that process positive samples and negative samples respectively.
Model distillation was first proposed by Hinton et al. in 2015; by having a small model learn the output distribution of a large model, the student model can often retain 80% to 95% of the teacher model's performance.
玻尔兹曼机由Hinton和Sejnowski在1985年提出，其状态概率分布遵循玻尔兹曼分布 P(x) ∝ exp(-E(x)/T)
知识蒸馏最早由Hinton等人在2015年提出，DiffusionGemma创造性地将其用于步数压缩，教师模型用较多步数（如48步）生成，学生模型用更少步数（如12步）达到相近质量
Knowledge Distillation is a technique for transferring knowledge from a large teacher model to a smaller student model, proposed by Geoffrey Hinton et al. in 2015
All Facts (20)
Knowledge Distillation is a technique for transferring knowledge from a large teacher model to a smaller student model, proposed by Geoffrey Hinton et al. in 2015
90%VerifiedThe backpropagation algorithm was systematized by Rumelhart, Hinton, and others in a paper published in Nature in 1986.
90%VerifiedKnowledge Distillation was originally proposed by Hinton et al. in 2015, with the core idea of having a small model learn the soft labels of a large model rather than hard labels.
90%VerifiedKnowledge distillation was proposed by Hinton et al. in 2015, originally referring to training a small model using the soft label outputs of a large model.
90%VerifiedKnowledge distillation was first proposed by Hinton et al. in 2015.
90%VerifiedThe core idea of knowledge distillation is to have the student model imitate the soft labels (probability distributions across categories) output by the teacher model, with soft labels being referred to by Hinton as "dark knowledge".
80%VerifiedThe backpropagation algorithm was popularized by Rumelhart, Hinton, and Williams in their 1986 paper, with the chain rule as its mathematical core.
80%Verified知识蒸馏由Hinton等人在2015年提出，核心思想是让小型学生模型模仿大型教师模型的输出分布
80%VerifiedThe MoE architecture was first proposed by Jacobs et al. in 1991 and in recent years has been brought back into the mainstream by models such as Google's Switch Transformer and Mixtral.
80%VerifiedIn 2012, Hinton's team's AlexNet won the ImageNet competition by an overwhelming margin.
80%VerifiedModel distillation was first proposed by Hinton et al. in 2015; by having a small model learn the output distribution of a large model, the student model can often retain 80% to 95% of the teacher model's performance.
75%VerifiedKnowledge distillation was first proposed by Hinton, Vinyals, and Dean at Google in 2015, with the core idea of having a small student model learn from the soft labels output by a large teacher model.
75%VerifiedKnowledge distillation was systematized by Hinton et al. in 2015; its core idea is to transfer the dark knowledge of a large model to a small model, using soft probability distributions as supervision signals.
75%VerifiedModel distillation technology was proposed by Geoffrey Hinton's team in 2015, originally used for model compression, by having a student model fit the soft-label outputs of a teacher model.
70%VerifiedThe Mixture of Experts (MOE) architecture was first proposed by Hinton et al. in 1991.
65%UnverifiedModel distillation is a model compression and knowledge transfer technique first proposed by Hinton et al. in 2015.
50%UnverifiedThe concept of fast weights can be traced back to the fast weight associative memory proposed by Hinton and von der Malsburg in the 1990s, as well as Schmidhuber's linear Transformer work.
50%UnverifiedIn 1986, Rumelhart, Hinton, and Williams published the paper "Learning representations by back-propagating errors" in the journal Nature.
50%UnverifiedThe paper's signatories include Anthropic co-founder Jack Clark and OpenAI's Chief Scientist.
50%UnverifiedHinton proposed the Forward-Forward Algorithm in 2022, replacing one forward pass plus one backward pass with two forward passes that process positive samples and negative samples respectively.
50%