Recurrent Architecture
A model architecture design that improves model quality without increasing parameter count by repeatedly feeding activations through the same set of layers, but whose inference-time KV cache usage grows linearly with loop depth.
Timeline (last 90 days)
Recurrent architectures improve model quality by feeding activations repeatedly through the same set of layers without increasing the parameter count.
In recurrent architectures, each loop independently generates a KV cache at inference time, so memory usage grows linearly with loop depth.
In a standard Transformer, each layer computes KV only once; whereas in a recurrent architecture the same set of layers is executed multiple times, so looping 4 times is equivalent to storing 4 copies of the KV cache.
KV-shareability is not stable in recurrent models: even with the same architecture and training method, some models have this property and some do not.
The idea of recurrent architecture can be traced back to the Universal Transformer and Deep Equilibrium Models, boosting reasoning ability by increasing computational depth without increasing the number of parameters.
All Facts (5)
Recurrent architectures improve model quality by feeding activations repeatedly through the same set of layers without increasing the parameter count.
50%UnverifiedIn recurrent architectures, each loop independently generates a KV cache at inference time, so memory usage grows linearly with loop depth.
50%UnverifiedIn a standard Transformer, each layer computes KV only once; whereas in a recurrent architecture the same set of layers is executed multiple times, so looping 4 times is equivalent to storing 4 copies of the KV cache.
50%UnverifiedKV-shareability is not stable in recurrent models: even with the same architecture and training method, some models have this property and some do not.
50%UnverifiedThe idea of recurrent architecture can be traced back to the Universal Transformer and Deep Equilibrium Models, boosting reasoning ability by increasing computational depth without increasing the number of parameters.
50%