Causal Encoder-Decoder
A hybrid Transformer architecture design that divides model layers into two components: a causal encoder and a decoder. The encoder processes only input tokens, while the decoder is used exclusively for output prediction. This separation reduces overall computational load and significantly compresses KV cache memory usage.
Timeline (last 90 days)
DeepSeek V4.1 Flash adopts a Causal Encoder-Decoder (CED) architecture, in which a 40-layer transformer has the first 20 layers as a causal encoder and the last 20 layers as a decoder.
In the CED architecture, the encoder runs only when processing input tokens and the decoder runs only when predicting output, meaning only half of the model weights need to be activated during the input phase.
In the CED architecture, the decoder's global KV cache is projected from the hidden states of the encoder's last layer, rather than from the hidden states of each individual layer.
The new DeepSeek model adopts an encoder-decoder hybrid architecture, splitting the 40-layer network in the middle, with the bottom 20 layers serving as the encoder and the top 20 layers as the decoder.
All Facts (4)
DeepSeek V4.1 Flash adopts a Causal Encoder-Decoder (CED) architecture, in which a 40-layer transformer has the first 20 layers as a causal encoder and the last 20 layers as a decoder.
50%UnverifiedIn the CED architecture, the encoder runs only when processing input tokens and the decoder runs only when predicting output, meaning only half of the model weights need to be activated during the input phase.
50%UnverifiedIn the CED architecture, the decoder's global KV cache is projected from the hidden states of the encoder's last layer, rather than from the hidden states of each individual layer.
50%UnverifiedThe new DeepSeek model adopts an encoder-decoder hybrid architecture, splitting the 40-layer network in the middle, with the bottom 20 layers serving as the encoder and the top 20 layers as the decoder.
50%