logprob
The natural logarithm of the raw probability that a language model assigns to each candidate token. It reflects the model's underlying probability distribution during generation and can be used for statistical confidence estimation.
Timeline (last 90 days)
logprob measures how fluent it is to generate that token, which is not equivalent to how correct the fact is; a model can perfectly well generate a false statement fluently and with high probability, so it remains a relative signal rather than an absolute guarantee.
logprob is log probability: when the token probability is 0.9, logprob is about -0.105, and when the probability is 0.01, it is about -4.605; the closer the value is to 0, the more confident the model is.
In practice, the logprobs of the tokens in an answer can be averaged or geometrically averaged to obtain sequence-level confidence, or the perplexity of the answer can be computed as an overall measure of certainty.
logprob thresholds need to be calibrated for the specific model and task distribution; directly applying a fixed threshold often works poorly.
The average logprob over an answer sequence can serve as a proxy for overall confidence, and self-consistency sampling assesses confidence by observing answer agreement across multiple samples.
A large language model's confidence signal can be extracted from the output probability distribution, for example by averaging the logprobs of candidate tokens or computing entropy.
In large model RL training, the logprobs computed by the training framework and those computed by the inference framework during the rollout phase often do not match, which gets amplified in the importance sampling weights of algorithms such as GRPO and PPO, leading to distorted training signals or even training collapse.
Logprob mismatch is one of the fundamental causes of instability and poor reproducibility in many RL training runs; after achieving bit-for-bit alignment, the determinism and reproducibility of training improve significantly.
All Facts (8)
logprob measures how fluent it is to generate that token, which is not equivalent to how correct the fact is; a model can perfectly well generate a false statement fluently and with high probability, so it remains a relative signal rather than an absolute guarantee.
50%Unverifiedlogprob is log probability: when the token probability is 0.9, logprob is about -0.105, and when the probability is 0.01, it is about -4.605; the closer the value is to 0, the more confident the model is.
50%UnverifiedIn practice, the logprobs of the tokens in an answer can be averaged or geometrically averaged to obtain sequence-level confidence, or the perplexity of the answer can be computed as an overall measure of certainty.
50%Unverifiedlogprob thresholds need to be calibrated for the specific model and task distribution; directly applying a fixed threshold often works poorly.
50%UnverifiedThe average logprob over an answer sequence can serve as a proxy for overall confidence, and self-consistency sampling assesses confidence by observing answer agreement across multiple samples.
50%UnverifiedA large language model's confidence signal can be extracted from the output probability distribution, for example by averaging the logprobs of candidate tokens or computing entropy.
50%UnverifiedIn large model RL training, the logprobs computed by the training framework and those computed by the inference framework during the rollout phase often do not match, which gets amplified in the importance sampling weights of algorithms such as GRPO and PPO, leading to distorted training signals or even training collapse.
50%UnverifiedLogprob mismatch is one of the fundamental causes of instability and poor reproducibility in many RL training runs; after achieving bit-for-bit alignment, the determinism and reproducibility of training improve significantly.
50%