Semantic Entropy
A method for measuring uncertainty in language model outputs. It semantically clusters multiple sampled answers before computing entropy, thereby filtering out spurious uncertainty introduced by lexical variation and more accurately reflecting the model's epistemic uncertainty.
Core Facts
Timeline (last 90 days)
The computational overhead of semantic entropy primarily stems from multiple forward inference samples and NLI judgments, which represent a non-negligible cost for compute-constrained edge scenarios.
Semantic entropy successfully recovers a usable confidence signal, filling the blind spot of token-level entropy on SLMs.
Semantic Entropy was originally proposed by Kuhn et al. in 2023 at NeurIPS.
Routing queries that semantic entropy judges as uncertain to a larger expert model for processing yields accuracy improvements of up to +50 percentage points.
The study concludes that the quality of the expert model is more important than architectural compatibility for routing gains.
The open-source hallucination detector's detection speed is approximately 90,000 times faster than the semantic entropy method.
The semantic entropy method requires multiple model inference calls, and latency can reach hundreds of milliseconds or even several seconds.
The semantic entropy method does not rely on the internal structure of a specific model; it infers uncertainty through semantic consistency at the output level, giving it relatively strong generalization ability.
A layered defense system can combine rapid screening (real-time preliminary filtering by a lightweight detector) with deep verification (secondary confirmation by heavyweight methods such as semantic entropy) to balance speed and accuracy.
The semantic entropy method is technically implemented by obtaining candidate answers through multiple samplings, using an NLI model to determine semantic equivalence for clustering, and then computing entropy over the semantic categories.
All Facts (10)
The semantic entropy method is technically implemented by obtaining candidate answers through multiple samplings, using an NLI model to determine semantic equivalence for clustering, and then computing entropy over the semantic categories.
85%UnverifiedA layered defense system can combine rapid screening (real-time preliminary filtering by a lightweight detector) with deep verification (secondary confirmation by heavyweight methods such as semantic entropy) to balance speed and accuracy.
60%UnverifiedThe computational overhead of semantic entropy primarily stems from multiple forward inference samples and NLI judgments, which represent a non-negligible cost for compute-constrained edge scenarios.
50%UnverifiedSemantic Entropy was originally proposed by Kuhn et al. in 2023 at NeurIPS.
50%UnverifiedRouting queries that semantic entropy judges as uncertain to a larger expert model for processing yields accuracy improvements of up to +50 percentage points.
50%UnverifiedThe study concludes that the quality of the expert model is more important than architectural compatibility for routing gains.
50%UnverifiedSemantic entropy successfully recovers a usable confidence signal, filling the blind spot of token-level entropy on SLMs.
50%UnverifiedThe semantic entropy method does not rely on the internal structure of a specific model; it infers uncertainty through semantic consistency at the output level, giving it relatively strong generalization ability.
50%UnverifiedThe semantic entropy method requires multiple model inference calls, and latency can reach hundreds of milliseconds or even several seconds.
50%UnverifiedThe open-source hallucination detector's detection speed is approximately 90,000 times faster than the semantic entropy method.
50%