Unverified60% confidenceFactExact time
Saxe and other researchers challenged the compression phase of the information bottleneck, finding that the compression phase is not universally present and its occurrence depends on the choice of activation function (e.g., compression often does not appear when using ReLU)
2
Sources
60%
Confidence
Long-term
Relevance
7/12/2026
First Seen
Sources
The Theory of Deep Learning: Why Do Neural Networks Work? A Complete Overview of Theoretical Research
hackernewshackernews7/11/2026
Related Claims
UnverifiedExcessive compression of prompts may introduce ambiguity, and models do not necessarily generalize well to overly unconventional compressed syntax.72% similarUnverifiedToken compression involves trade-offs: some seemingly redundant contextual information may be critical for model reasoning, and aggressive compression may affect output quality in scenarios such as code generation and complex reasoning.71% similarUnverifiedThe choice of bottleneck dimensionality is essentially a trade-off between information compression rate and reconstruction quality, corresponding to rate-distortion theory; Shannon proved in 1959 that there exists a theoretical minimum coding rate lower bound for a given distortion tolerance.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/494300API
curl https://kongchang.com/api/v1/knowledge/claims/494300MCP
get_claim(id=494300)