Cross-Entropy Loss
A loss function used to measure the difference between probability distributions, commonly applied in classification and language model training. In language models it computes the difference between the predicted probability distribution over the next token and the true distribution, imposing a larger penalty on confident incorrect predictions.
Timeline (last 90 days)
SFT是一种用成对输入输出和交叉熵损失更新模型参数的对齐方法，不依赖奖励模型或偏好比较。
Conventional training (such as cross-entropy loss) only penalizes incorrect predictions without constraining probability calibration quality, causing models to be overconfident or overly conservative.
The joint training of this method uses only cross-entropy loss, simultaneously accounting for both accuracy and calibration quality, at the cost of introducing only one additional hyperparameter: the calibrator learning rate.
The cross-entropy loss optimized during modern large language model training is essentially minimizing the encoding length of text.
Cross-entropy loss is the most commonly used loss function for classification tasks; rooted in information theory, it has gradient properties better suited to classification problems than MSE.
Cross-entropy loss is the most commonly used loss function in classification tasks, rooted in information theory, and measures the distance between the model's predicted probability distribution and the true distribution.
The project models Tic-Tac-Toe as a supervised classification problem, with the encoded board state as input and the probability distribution of the best move among the 9 cells as output, trained using cross-entropy loss.
All Facts (7)
Cross-entropy loss is the most commonly used loss function for classification tasks; rooted in information theory, it has gradient properties better suited to classification problems than MSE.
60%UnverifiedCross-entropy loss is the most commonly used loss function in classification tasks, rooted in information theory, and measures the distance between the model's predicted probability distribution and the true distribution.
60%UnverifiedSFT是一种用成对输入输出和交叉熵损失更新模型参数的对齐方法，不依赖奖励模型或偏好比较。
50%UnverifiedConventional training (such as cross-entropy loss) only penalizes incorrect predictions without constraining probability calibration quality, causing models to be overconfident or overly conservative.
50%UnverifiedThe joint training of this method uses only cross-entropy loss, simultaneously accounting for both accuracy and calibration quality, at the cost of introducing only one additional hyperparameter: the calibrator learning rate.
50%UnverifiedThe cross-entropy loss optimized during modern large language model training is essentially minimizing the encoding length of text.
50%UnverifiedThe project models Tic-Tac-Toe as a supervised classification problem, with the encoded board state as input and the probability distribution of the best move among the 9 cells as output, trained using cross-entropy loss.
50%