Unverified50% confidenceFactExact time
Cactus replaces the feed-forward layer of the traditional Transformer with a headband MLP, containing only approximately 25,600 parameters per layer instead of 4.7 million, with most knowledge stored in an engram table that retrieves short 2-to-3-word fragments via hash lookup.
1
Sources
50%
Confidence
Long-term
Relevance
10/5/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedTransformer注意力机制的计算量随序列长度平方增长，早期语言模型上下文窗口上限约4000个token64% similarUnverifiedTransformer推理中批量大小为1时注意力层的算术强度约1~10 FLOP/Byte，远低于H100约300 FLOP/Byte的屋脊点，属于带宽受限场景63% similarUnverified标准Transformer自注意力计算量随序列长度平方增长，50万token的注意力计算量约为4096token的15000倍61% similarUnverifiedThe combination of Pythia's first 5 layers plus RWKV's remaining 27 layers reduced the Transformer's KV cache by 84.4%, with no significant difference in accuracy compared to using RWKV alone.61% similarUnverified批大小为1的Transformer推理算术强度往往低于10 FLOP/Byte，意味着GPU算力利用率不足理论峰值的5%59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/977850API
curl https://kongchang.com/api/v1/knowledge/claims/977850MCP
get_claim(id=977850)