SelfAttention
An attention mechanism that allows a model to attend to different positions within a sequence when processing that sequence.
Timeline (last 90 days)
In terms of network structure, a CNN processes a 21x21 local observation window, an LSTM carries memory across steps, and three attention modules—SelfAttention, MemoryCrossAttention and FiLM—are stacked on top of the CNN.
Ablation experiments show that FiLM does almost all of the effective work, while SelfAttention and MemoryCrossAttention contribute close to zero in the trained model.
All Facts (2)
In terms of network structure, a CNN processes a 21x21 local observation window, an LSTM carries memory across steps, and three attention modules—SelfAttention, MemoryCrossAttention and FiLM—are stacked on top of the CNN.
50%UnverifiedAblation experiments show that FiLM does almost all of the effective work, while SelfAttention and MemoryCrossAttention contribute close to zero in the trained model.
50%