Cross-Entropy
A quantity in information theory that measures the difference between two probability distributions. In LLM decoding it is used as a constraint to limit the degree to which sampling deviates from the correct answer.
Timeline (last 90 days)
The article recommends replacing standard softmax and cross-entropy implementations in custom training loops to avoid numerical issues.
The authors' fixed implementations of log-softmax, softmax, and cross-entropy add no computational complexity.
The structural root of the LLM calibration problem is that the pretraining objective is to minimize the cross-entropy of the next word rather than to output trustworthy probability estimates, and RLHF alignment further amplifies the tendency to express things as if certain.
Cross-entropy is a per-token average metric and does not additionally penalize the structural defect that some tokens are never processed at all, so when comparing routing strategies one cannot look only at a single average evaluation loss.
The forward propagation process of logistic regression is: linear combination z = w·x + b, activated through sigmoid as ŷ = 1/(1+e^(-z)), followed by cross-entropy loss calculation.
With sigmoid activation combined with cross-entropy loss, differentiating the loss with respect to the weights simplifies to ∂L/∂w = (ŷ - y) · x.
MSE suffers from gradient saturation when Sigmoid outputs are close to 0 or 1, causing training to be extremely slow, whereas cross-entropy has larger gradients when predictions deviate severely
该项目将博弈问题建模为监督分类任务，神经网络输入为编码后的棋盘状态，输出为9个格子中最佳落子位置的概率分布，通过交叉熵损失学习模仿Minimax决策
交叉熵损失函数 H(p,q) = -Σp(x)log q(x) 的最小值在q=p时取到，等于真实分布的熵
当模型对某个训练样本的预测概率接近1时，该样本对应的交叉熵损失趋近于零，模型几乎无法获得梯度更新信号
1 more timeline events
All Facts (11)
The article recommends replacing standard softmax and cross-entropy implementations in custom training loops to avoid numerical issues.
50%UnverifiedThe authors' fixed implementations of log-softmax, softmax, and cross-entropy add no computational complexity.
50%UnverifiedThe structural root of the LLM calibration problem is that the pretraining objective is to minimize the cross-entropy of the next word rather than to output trustworthy probability estimates, and RLHF alignment further amplifies the tendency to express things as if certain.
50%UnverifiedCross-entropy is a per-token average metric and does not additionally penalize the structural defect that some tokens are never processed at all, so when comparing routing strategies one cannot look only at a single average evaluation loss.
50%UnverifiedThe forward propagation process of logistic regression is: linear combination z = w·x + b, activated through sigmoid as ŷ = 1/(1+e^(-z)), followed by cross-entropy loss calculation.
50%UnverifiedWith sigmoid activation combined with cross-entropy loss, differentiating the loss with respect to the weights simplifies to ∂L/∂w = (ŷ - y) · x.
50%UnverifiedMSE suffers from gradient saturation when Sigmoid outputs are close to 0 or 1, causing training to be extremely slow, whereas cross-entropy has larger gradients when predictions deviate severely
50%Unverified该项目将博弈问题建模为监督分类任务，神经网络输入为编码后的棋盘状态，输出为9个格子中最佳落子位置的概率分布，通过交叉熵损失学习模仿Minimax决策
50%Unverified交叉熵损失函数 H(p,q) = -Σp(x)log q(x) 的最小值在q=p时取到，等于真实分布的熵
50%Unverified当模型对某个训练样本的预测概率接近1时，该样本对应的交叉熵损失趋近于零，模型几乎无法获得梯度更新信号
50%Unverified监督学习本质上是函数逼近问题，通过最小化损失函数实现，常见损失函数包括回归任务的均方误差（MSE）和分类任务的交叉熵（Cross Entropy）
50%