Transformer
Transformer is a deep learning model architecture based on the self-attention mechanism, proposed by Google in 2017. It processes sequence data in parallel and captures long-range dependencies, overcoming the limitations of traditional recurrent neural networks. The architecture is widely applied in natural language processing, computer vision, speech recognition, and other domains, and serves as the foundational structure for large-scale pretrained language models such as BERT and the GPT series.
Core Facts
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
The self-attention mechanism maps inputs into three sets of vectors—Query, Key, and Value—computes dot-product similarity followed by softmax normalization, and then performs a weighted sum of the Values to generate a representation that incorporates global context.
Timeline (last 90 days)
Examples of technical-route debates include: in model architecture, the debate between Transformer and hybrid architectures that alternate convolution and attention; in training methods, the trade-off between self-supervised learning and supervised learning; in deployment strategy, the weighing of cloud large models against on-device small models.
Mainstream Transformers use the self-attention mechanism, and the computation and memory usage of the standard implementation grow quadratically with sequence length.
Since it was proposed, Mamba has been regarded as one of the few approaches with the potential to compete head-on with Transformer in general sequence modeling.
Even if Mamba passes validation, what is more likely in the short term is a hybrid approach—introducing state space layers at specific stages rather than replacing Transformer wholesale.
Around Transformer, extremely mature inference frameworks, operator libraries, hardware adaptations, and tuning experience have already accumulated.
For a new architecture to obtain the same degree of engineering support as Transformer, framework developers need to implement custom operators, hardware vendors need to provide adaptations, and extensive tuning on real workloads is required—a process typically measured in years.
The typical technical path for video-to-game generation includes: first extracting objects, actions, and scene graphs through a video understanding model (such as a Transformer-based video encoder), then building the game state machine, collision detection, and interaction logic through a rule engine or generative model.
Representative generative model technologies include generative adversarial networks (GANs), variational autoencoders (VAEs) and Transformer-architecture large language models.
Taking a Transformer as an example, the output of each neuron or attention head can be viewed as a direction in a high-dimensional space, and changing the projection strength along that direction systematically affects downstream computation.
The NCM model contains a compressor (which encodes text into multi-layer concept codes) and a Transformer generator (which predicts the next layer of concepts), with one forward pass per layer.
40 more timeline events
All Facts (20)
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
90%VerifiedKV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedThe self-attention mechanism maps inputs into three sets of vectors—Query, Key, and Value—computes dot-product similarity followed by softmax normalization, and then performs a weighted sum of the Values to generate a representation that incorporates global context.
90%VerifiedThe Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
90%VerifiedMainstream LLMs are based on the Transformer architecture proposed by Google in 2017, whose core innovation is the Self-Attention mechanism.
90%VerifiedThe Prefill phase is a compute-intensive operation, while the Decode phase is a memory-bandwidth-intensive operation.
90%VerifiedThe VRAM occupied by KV Cache grows linearly with sequence length.
90%VerifiedWhen large language models process long contexts, the computational complexity of the attention mechanism grows proportionally to the square of the sequence length.
90%VerifiedThe Vision Transformer (ViT) was proposed by Google in 2020, dividing images into fixed-size patches as sequence inputs to a Transformer encoder.
90%VerifiedThe self-attention mechanism of the standard Transformer architecture has a computational complexity of O(n²).
90%VerifiedThe Transformer architecture was proposed by Google in 2017.
90%VerifiedLarge language models are based on the Transformer architecture and are trained on massive amounts of text data.
90%VerifiedIn the Transformer architecture, the tokens of system instructions are placed at the very front of the attention mechanism, exerting a continuous constraining influence on all subsequently generated content.
90%VerifiedThe key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
90%VerifiedThe Transformer architecture was proposed by Google in the 2017 paper "Attention Is All You Need."
90%VerifiedThe computational complexity of the standard Self-Attention mechanism has a quadratic relationship with sequence length (O(n²)).
90%VerifiedThe Transformer architecture was proposed in 2017 by the Google Brain team in the paper "Attention Is All You Need"
90%VerifiedWhisper is an automatic speech recognition model open-sourced by OpenAI in 2022, built on a Transformer architecture and trained on 680,000 hours of multilingual audio data.
90%VerifiedVision Transformer (ViT) was proposed by the Google Brain team in 2020, introducing the Transformer architecture into the field of computer vision.
90%VerifiedThe core working principle of large language models such as GPT-4, Claude, and Gemini is next-token prediction based on the Transformer architecture.
90%