RNN
A Recurrent Neural Network (RNN) is a class of deep learning models designed for processing sequential data. Its defining characteristic is recurrent connections between neurons, which propagate hidden states across time steps to capture contextual dependencies within sequences. RNNs are widely applied in natural language processing, speech recognition, and time-series forecasting. Basic RNNs suffer from vanishing and exploding gradient problems, which led to the development of improved variants such as LSTM (Long Short-Term Memory) and GRU for more effective modeling of long-range dependencies.
Core Facts
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
The key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
Through the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
Timeline (last 90 days)
Recurrent models must be trained on shuffled, independent-window segments; if streaming windows are fed in chronological order, the network collapses to predicting only one dominant gesture.
In backpropagation through time (BPTT), the longer the sequence, the more easily the repeated multiplication of gradients along the time dimension leads to explosion or vanishing, whereas independent windows cut off such long-range dependencies and instead improve robustness in cross-user scenarios.
RNNs suffer from the long-range dependency problem: as sequences grow longer, early information tends to suffer vanishing gradients during backpropagation.
The Transformer architecture replaced the recurrent structure of RNNs, greatly improving training efficiency, and is the underlying backbone of modern large language models such as GPT and BERT.
Each Transformer block contains a self-attention layer and a position-wise feed-forward network, both surrounded by residual connections and layer normalization.
The long-range dependencies of RNNs decay, so by the time early information is passed along, little of it remains.
SNACK enables sequence models (such as Transformers and RNNs) to directly handle graph generation tasks without a complex graph neural network encoder.
Classic experiments show that a plain RNN is almost unable to correctly capture dependencies when the gap exceeds about 10-20 words.
Models in the RNN era read word by word strictly from left to right without looking back, causing severe decay of contextual information as the hidden state advances, i.e., the long-term dependency problem.
The self-attention mechanism captures dependencies of arbitrary distance in a single forward pass by dynamically computing association weights between an element and all other elements in the sequence, overcoming the long-range dependency difficulties caused by vanishing gradients in RNNs/LSTMs.
30 more timeline events
All Facts (20)
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
90%VerifiedThe key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
90%VerifiedThrough the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
85%VerifiedThe self-attention mechanism is capable of capturing dependencies between any two positions in a sequence.
80%VerifiedThe self-attention mechanism allows the model to simultaneously reference the contextual information of all other tokens in a sequence when processing each token, rather than processing sequentially as RNNs do
80%VerifiedThe self-attention mechanism of the Transformer allows the model to directly compute dependencies between any two positions in a sequence, breaking through the parallelization bottleneck caused by the sequential computation of RNNs/LSTMs.
80%VerifiedRNNs suffer from vanishing/exploding gradient problems, where gradients propagating across a large number of time steps decay or grow exponentially, making it difficult to learn long-range dependencies.
75%VerifiedDavid Ha and Schmidhuber published the paper "World Models" in 2018, demonstrating how an agent can complete reinforcement learning training within dreams
75%VerifiedEach Transformer layer contains two core sub-modules: multi-head self-attention and a feed-forward neural network.
75%VerifiedTransformer achieves efficient modeling of long texts through its self-attention mechanism.
70%VerifiedRNNs are designed specifically for sequential data and pass information between time steps through hidden states.
70%VerifiedThe Transformer architecture was proposed in 2017 and has largely supplanted RNNs in the NLP domain.
70%VerifiedBefore the advent of Transformer, the field of natural language processing relied primarily on RNNs and LSTMs to process sequential data, architectures that had to process text word by word in sequence and could not perform parallel computation
70%VerifiedRNNs prior to the Transformer could only process words sequentially one at a time, whereas the Transformer can process entire sequences in parallel, greatly improving training efficiency.
65%VerifiedCompared to traditional recurrent neural networks (RNNs), which tend to forget earlier content when processing long texts, Transformers can analyze multiple parts of a text simultaneously.
65%VerifiedIn the deep learning era, end-to-end models based on CNN and RNN (such as the CRNN architecture) significantly improved OCR recognition capabilities, and in recent years the Transformer architecture has also been introduced into the OCR field.
65%VerifiedTransformer的核心创新是用自注意力机制替代RNN的时序处理方式，实现了高度并行化训练，但放弃了RNN的隐状态跨时间步传递机制
65%VerifiedSet Transformer基于自注意力机制建模行向量交互，通过去掉位置编码保持排列不变性
65%VerifiedBecause each time step's output in an RNN depends on the hidden state of the previous step, RNNs cannot perform parallel computation and are unable to leverage the parallel processing advantages of GPUs.
65%VerifiedLSTM solves the vanishing gradient problem that standard RNNs encounter when processing long sequences by introducing three control mechanisms: the forget gate, the input gate, and the output gate
65%