Positional Encoding

Positional encoding was first widely used in the Transformer architecture, and its role is to inject the order information of elements in a sequence into the model, because the pure attention mechanism itself is insensitive to order.

Because the Transformer self-attention mechanism exhibits positional bias, earlier input content has a stronger anchoring effect on subsequent generation, which is the underlying reason for the context anchoring effect of system prompts

Hand-computing the 6×512 positional encoding table helps one understand how the combination of sine and cosine lets the model distinguish different positions without learning a separate vector for each position.

For Transformers, mechanisms such as positional encoding and attention masking may fail beyond the training length.

DDPM adopts the sinusoidal positional encoding from the Transformer, converting the time step t into a high-dimensional vector, then mapping it through two fully connected layers to a dimension matching the number of feature map channels, and finally modulating the features of each layer by element-wise addition or FiLM.

In the diffusion model, the time step t is converted into a vector via sine-cosine encoding similar to Transformer positional encoding, and then fused with the image features.

Tiny models like DinoGPT typically use learnable positional embeddings, with the parameter count equal to the maximum sequence length multiplied by the hidden dimension.

Positional encoding should be added to the embedding, not concatenated.

The context window is the maximum number of tokens a time-series foundation model (and all Transformer-based models) can process in a single forward computation, jointly determined by the positional encoding in the model architecture and the complexity of the attention mechanism; the window size is a hyperparameter fixed at training time and cannot be extended at inference.

The performance decline caused by 'lost in the middle' follows a U-shaped curve rather than linear decay; its mechanism may be related to Transformer attention allocation and positional encoding.

The causes of the lost-in-the-middle phenomenon are usually attributed to the decay effect of positional encoding, the fact that beginning and end content in training data is more often associated with answers, and the dilution of the attention softmax over long sequences.

STCNet adopts a dual-domain collaborative strategy, introducing physics-driven positional encoding in the projection domain and temporal encoding in the latent space, jointly optimizing the projection domain and the image domain.

Traditional sinusoidal/cosine positional encodings are designed for uniform integer step sizes and have difficulty expressing geometric relationships at irregular intervals.

Positional encoding is used to inject word order information into the model, because the attention mechanism itself does not distinguish positional order.

Positional Encoding solves the problem of the attention mechanism being inherently orderless and unaware of word sequence by attaching positional information to each word.

Transformer uses positional encoding based on sine and cosine functions, with wavelengths arranged in a geometric progression, where the shortest and longest differ by a factor of ten thousand.

The vector input to the model equals the word embedding plus the positional embedding; for example, 7 characters generate 7 word embeddings and 7 positional embeddings that are added together to produce the actual input

位置编码经历了从固定正弦编码（原始论文）到可学习绝对位置编码（BERT）再到旋转位置编码RoPE（LLaMA、Qwen）的演进