Softmax

Self-attention weights are normalized through the Softmax function, and the sum of the weights is always 1.

The core operation of Self-Attention is to compute the dot-product similarity between query vectors and key vectors, then use this as weights to perform a weighted sum of the value vectors.

The softmax operation is prone to numerical overflow or underflow due to insufficient precision when handling extreme values, which is the core technical reason why early FP8 quantization performed poorly.

MBCF uses Sigmoid rather than Softmax to generate modality weights, in order to avoid zero-sum competition between modalities.

The Softmax function converts the raw scores of multiple neurons in the output layer into a probability distribution: softmax(z_i) = e^{z_i} / Σ_j e^{z_j}, and together with cross-entropy loss it is commonly used for multi-class tasks.

For applications that need probabilities to decide whether to act, calibration checks are indispensable; Softmax only guarantees that probabilities sum to one, not that the probabilities' meaning is accurate.

The evaluation used a linear classifier with Softmax output, and ensured that the signal-level data splits were completely consistent.

Recommendations for choosing hidden layer activation functions: start with ReLU, and if a large number of neurons die, try Leaky ReLU; use Sigmoid for binary classification output; use Softmax for multi-class output; use no activation function for regression output.

Softmax converts a set of outputs into a probability distribution that sums to 1, and is used in the last layer of multi-class tasks, not in hidden layers.

Softmax tends to produce sharp, winner-takes-all distributions, whereas the distribution shape of squared normalization may be more spread out or more concentrated.

Attempts to replace Softmax with ReLU (ReLUformer) can train, but require additional normalization tricks to stabilize.

Dot products can be negative; Softmax maps them to small positive numbers via exponentiation, whereas taking the squared modulus causes positive and negative inner products to produce symmetric weight contributions.

The squared modulus in the Born rule is naturally non-negative, and after normalization automatically satisfies the probability distribution constraint, without needing to rely on an exponential function to enforce non-negativity as Softmax does.

A classification head is typically a simple fully connected layer with Softmax activation that maps a high-dimensional feature vector to a probability distribution over each category.

In the Decoder scenario, a Mask matrix must be introduced, assigning negative infinity to positions after the current word, which approach zero after Softmax, producing a causal generation effect.

During VQ training, the tau parameter is gradually annealed, and Softmax is used to ensure all quantization tiers continuously receive gradients.

In the standard Transformer architecture, the Softmax operation is responsible for normalizing attention scores into a probability distribution, determining the degree to which each token attends to other tokens

The nonlinear normalization and competitive attention allocation provided by Softmax is one of the important reasons for the superior performance of Transformers

Static Contraction refers to using pre-determined, structured tensor contraction operations to replace or circumvent runtime dynamic Softmax normalization

Softmax involves exponential operations, row-wise normalization, and global dependencies, making it difficult to parallelize and quantize, and it is especially resource-intensive in long-sequence scenarios