Verified65% confidenceFactTime unknown
The core objective of pre-training is to give the model the ability to predict the next token (Predict Next Token), based on the causal attention mechanism in the Transformer architecture.
3
Sources
65%
Confidence
Long-term
Relevance
6/3/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedTransformer 在预训练阶段通过 Next Token Prediction 任务学习概率分布，每次输出本质是在给定上文条件下预测下一个词最可能是什么85% similarUnverifiedLLM的预训练本质是在海量文本上进行下一词预测（Next Token Prediction），通过Transformer架构的注意力机制建立词语间语义关联82% similarUnverifiedThe hallucination phenomenon in large models is rooted in the autoregressive generation mechanism of Transformers, where the model learns probability distributions through the Next Token Prediction task during the pre-training phase.82% similarUnverifiedScheduled Sampling is a compromise approach to mitigating exposure bias, gradually introducing the model's own predictions as input during the later stages of training.77% similarUnverified基于Transformer的SAKT、AKT等模型引入注意力机制来建模历史练习对当前知识状态的影响，提升了预测精度75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/39839API
curl https://kongchang.com/api/v1/knowledge/claims/39839MCP
get_claim(id=39839)