GRU
A classic recurrent neural network variant that uses gating mechanisms to mitigate the vanishing gradient problem. Its structure is simpler than that of LSTM, and it is widely used in sequence modeling tasks.
Timeline (last 90 days)
GRU is a simplified version of LSTM, merging the three gates into two—the update gate and the reset gate—with fewer parameters and faster training, achieving results close to LSTM on many tasks.
Both LSTM and GRU retain a structure that computes sequentially by time step, and therefore cannot be parallelized.
Partial observability means a single agent cannot know the complete state of the environment and can only see local information; in multi-agent systems this must be modeled as a Dec-POMDP, and recurrent networks such as GRUs are often used to maintain temporal memory.
The recurrent state space model (RSSM) consists of a deterministic recurrent path (GRU) and a stochastic path (latent variables): the deterministic state h_t is updated by the previous moment's h_{t-1} and action a_{t-1} through an RNN, and the stochastic state z_t is sampled from a prior distribution parameterized by h_t; this hybrid design balances long-range memory and multimodal uncertainty expression.
LSTMs and GRUs alleviate the vanishing gradient problem by introducing gating mechanisms (forget gate, input gate, output gate), allowing the model to determine the pace of state updates on its own.
Early seq2seq implementations were mostly based on LSTM or GRU, while modern implementations extensively adopt the attention mechanism in the Transformer architecture, which handles long-range dependencies more effectively.
Recurrent network architectures (such as LSTM/GRU) endow agents with the ability to remember historical observations by maintaining a hidden state that is updated across time steps, thereby enabling coherent decision-making in partially observable environments.
The core pain point of traditional RNNs (including GRU and LSTM) is that historical information decays at an exponential rate during backpropagation, leading to vanishing gradients.
PLUG and ILRM were trained only on 16–64 step delays, successfully extrapolating to 4096 steps, achieving 100% accuracy on a one-bit information retention task, while the baseline standard GRU degraded to random-guess level.
On 1024-step sequences, the average gradient of the initial token for PLUG and ILRM remains stably at around the 10⁻⁴ order of magnitude, while the standard GRU gradient drops as low as approximately 10⁻¹⁴⁵, resulting in numerical underflow.
6 more timeline events
All Facts (17)
GRU (Gated Recurrent Unit) was proposed by Cho et al. in 2014, controlling information flow through two gating mechanisms—the update gate and the reset gate—with fewer parameters and faster training compared to LSTM.
60%UnverifiedTraditional RNNs suffer from the vanishing gradient problem; in practice, LSTMs and GRUs are more commonly used to address this.
60%UnverifiedBoth LSTM and GRU retain a structure that computes sequentially by time step, and therefore cannot be parallelized.
50%UnverifiedGRU is a simplified version of LSTM, merging the three gates into two—the update gate and the reset gate—with fewer parameters and faster training, achieving results close to LSTM on many tasks.
50%UnverifiedPartial observability means a single agent cannot know the complete state of the environment and can only see local information; in multi-agent systems this must be modeled as a Dec-POMDP, and recurrent networks such as GRUs are often used to maintain temporal memory.
50%UnverifiedThe recurrent state space model (RSSM) consists of a deterministic recurrent path (GRU) and a stochastic path (latent variables): the deterministic state h_t is updated by the previous moment's h_{t-1} and action a_{t-1} through an RNN, and the stochastic state z_t is sampled from a prior distribution parameterized by h_t; this hybrid design balances long-range memory and multimodal uncertainty expression.
50%UnverifiedLSTMs and GRUs alleviate the vanishing gradient problem by introducing gating mechanisms (forget gate, input gate, output gate), allowing the model to determine the pace of state updates on its own.
50%UnverifiedEarly seq2seq implementations were mostly based on LSTM or GRU, while modern implementations extensively adopt the attention mechanism in the Transformer architecture, which handles long-range dependencies more effectively.
50%UnverifiedRecurrent network architectures (such as LSTM/GRU) endow agents with the ability to remember historical observations by maintaining a hidden state that is updated across time steps, thereby enabling coherent decision-making in partially observable environments.
50%UnverifiedPLUG and ILRM were trained only on 16–64 step delays, successfully extrapolating to 4096 steps, achieving 100% accuracy on a one-bit information retention task, while the baseline standard GRU degraded to random-guess level.
50%UnverifiedOn 1024-step sequences, the average gradient of the initial token for PLUG and ILRM remains stably at around the 10⁻⁴ order of magnitude, while the standard GRU gradient drops as low as approximately 10⁻¹⁴⁵, resulting in numerical underflow.
50%UnverifiedThe core pain point of traditional RNNs (including GRU and LSTM) is that historical information decays at an exponential rate during backpropagation, leading to vanishing gradients.
50%UnverifiedRNNs have two major weaknesses: vanishing gradients and the inability to perform parallel computation; LSTM and GRU alleviate the vanishing gradient problem through gating mechanisms.
50%UnverifiedLSTM and GRU alleviate the long-range dependency problem of standard RNNs through gating mechanisms.
50%Unverified解决POMDP的经典方法是维护信念状态（belief state），但在高维空间中计算上不可行；现代深度强化学习通过LSTM或GRU等循环结构隐式维护历史信息
50%Unverified现代深度强化学习通过在策略网络中引入循环结构（如LSTM或GRU）来隐式维护历史信息以处理POMDP
50%Unverified在Transformer出现之前，主流的序列建模方法是循环神经网络（RNN）及其变体LSTM和GRU
50%