Unverified50% confidenceFactExact time
Entropy and KL divergence in information theory are important foundations for understanding loss function design and generative models (such as VAEs and diffusion models)
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedCross-entropy loss is the most commonly used loss function in classification tasks, rooted in information theory, and measures the distance between the model's predicted probability distribution and the true distribution.77% similarUnverifiedKL散度在变分推断（VAE的ELBO目标）、知识蒸馏和模型压缩中扮演核心角色73% similarUnverifiedThe concept of perplexity originates from entropy (Shannon Entropy) in information theory, and is essentially the average branching factor per prediction step of a language model.71% similarVerifiedThe semantic entropy method is technically implemented by obtaining candidate answers through multiple samplings, using an NLI model to determine semantic equivalence for clustering, and then computing entropy over the semantic categories.71% similarUnverified交叉熵损失函数鼓励模型将正确类别的logit推向无穷大，导致深度网络预测过于确定70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/904270API
curl https://kongchang.com/api/v1/knowledge/claims/904270MCP
get_claim(id=904270)