Recurrent Neural Network
A class of neural network architectures that process sequential data through recursive updates of a hidden state. They are adept at capturing contextual dependencies along the temporal dimension and are widely used in sequence modeling tasks.
Timeline (last 90 days)
Sub-quadratic loop alternatives (such as RNNs and SSMs) have compute that grows linearly with sequence length (O(n·d)), but limited state capacity may degrade long-range retrieval quality.
Recurrent compressed memory is a mechanism in long-context LLMs that compresses historical information into a fixed-size recurrent state (such as a recurrent neural network hidden state or a memory matrix), whose memory footprint does not grow with context length.
In animal communication research, commonly used machine learning methods include unsupervised clustering, supervised classification and sequence modeling (such as hidden Markov models or recurrent neural networks).
In recurrent network auditing, if H_t is the hidden state at time t and X_t is the current input, then I(H_t; X_t) characterizes the transient component, while I(H_t; θ) or I(H_t; training distribution) characterizes the durable component carried by the weights.
In recurrent networks, the catastrophic forgetting problem is amplified: because hidden states are recursively coupled over time, a small change in the weights may be amplified exponentially through multi-step propagation.
The study finds that the output geometries of Transformers, state space models, and recurrent neural networks are aligned to a far greater degree than their activation geometries.
In recurrent neural networks, the critical state corresponds to the maximum singular value (spectral radius) of the Jacobian matrix approaching 1.
A recurrent neural network containing 417,000 parameters is capable of autonomously generating the complete 6,500-frame Bad Apple video from a single initial state.
经验回放将交互经验存入固定大小缓冲区并随机采样小批量训练，打破连续经验的时间相关性，使神经网络训练更稳定
All Facts (9)
Sub-quadratic loop alternatives (such as RNNs and SSMs) have compute that grows linearly with sequence length (O(n·d)), but limited state capacity may degrade long-range retrieval quality.
50%UnverifiedRecurrent compressed memory is a mechanism in long-context LLMs that compresses historical information into a fixed-size recurrent state (such as a recurrent neural network hidden state or a memory matrix), whose memory footprint does not grow with context length.
50%UnverifiedIn animal communication research, commonly used machine learning methods include unsupervised clustering, supervised classification and sequence modeling (such as hidden Markov models or recurrent neural networks).
50%UnverifiedIn recurrent network auditing, if H_t is the hidden state at time t and X_t is the current input, then I(H_t; X_t) characterizes the transient component, while I(H_t; θ) or I(H_t; training distribution) characterizes the durable component carried by the weights.
50%UnverifiedIn recurrent networks, the catastrophic forgetting problem is amplified: because hidden states are recursively coupled over time, a small change in the weights may be amplified exponentially through multi-step propagation.
50%UnverifiedThe study finds that the output geometries of Transformers, state space models, and recurrent neural networks are aligned to a far greater degree than their activation geometries.
50%UnverifiedIn recurrent neural networks, the critical state corresponds to the maximum singular value (spectral radius) of the Jacobian matrix approaching 1.
50%UnverifiedA recurrent neural network containing 417,000 parameters is capable of autonomously generating the complete 6,500-frame Bad Apple video from a single initial state.
50%Unverified经验回放将交互经验存入固定大小缓冲区并随机采样小批量训练，打破连续经验的时间相关性，使神经网络训练更稳定
50%