Information Theory

Claude Shannon published 'A Mathematical Theory of Communication' in 1948, laying the foundation for modern information theory.

The concept of information entropy originates from information theory, proposed by Claude Shannon in 1948 to measure uncertainty and disorder in information systems.

Historically, mathematical frameworks such as information theory and cybernetics laid the foundation for subsequent technological breakthroughs.

Under the framework of information theory, the information content of a message depends on how much uncertainty it eliminates; the more unpredictable a message is, the greater its information content.

Generative models cannot produce information beyond their learned parameters; every synthetic generation introduces sampling variance and truncates rare tokens, causing downstream training to accumulate statistical bias, and the recursive loop is essentially a process of information loss.

Surprisal is a fundamental concept in information theory, referring to how unexpected a word or string is in a given context, commonly measured by negative log probability: the lower the probability, the higher the surprisal.

The design of decoding strategies is moving from experience-driven to theory-driven, and information theory can provide a more solid theoretical foundation and clearer performance bounds.

The direction EOS represents is to re-derive decoding heuristics using a rigorous information-theoretic and online learning framework, and to provide provable performance bounds.

In the intersection of information theory and verification, formalization on such a large scale is rare, and this paper sets a high standard for the reliability of its conclusions.

This framework uses information theory to decompose reconstruction difficulty into two orthogonal sources: information leaked during training and the inherent uncertainty of the data itself.

The foundational paper in information theory was completed by a single author, and almost no computer science paper since has surpassed it in impact.

Channel capacity degradation in information theory can describe the model collapse process: each time a signal undergoes lossy transmission and is re-encoded, quantization errors are introduced, and if the output is used as the next input, errors accumulate exponentially.

In his seminal 1948 paper, Shannon noted that closed-loop self-referential systems will ultimately converge to a degenerate low-entropy state in the absence of an external correction signal.

Compression and prediction are essentially the same thing; a model that compresses data well must have a good ability to model the data distribution.

Entropy and KL divergence in information theory are important foundations for understanding loss function design and generative models (such as VAEs and diffusion models)

The binary encoding strategy achieves the information-theoretic optimal lower bound of log₂(n).

Shannon pointed out that if an event has n equally probable possible outcomes, the minimum amount of information needed to determine the specific outcome is log₂(n) bits.

The binary encoding strategy for the poison bottle problem exactly achieves the information-theoretic lower bound of log₂(n), making it an information-theoretically optimal solution.

A rat has two states—alive or dead—which is equivalent to carrying 1 bit of information.

log₂(n) is the minimum amount of information needed to distinguish n objects, which reflects information-theoretic thinking.