decoder-only
A model architecture that uses only the Transformer decoder stack, predicting the next token in an autoregressive manner, suited to conversational generation tasks.
Timeline (last 90 days)
The decoder-only architecture uses only the Transformer decoder stack, predicting the next token one by one in an autoregressive manner through a causal attention mask, learning during training to predict subsequent content based on preceding context.
In a decoder-only architecture, the causal mask ensures each token can only see itself and earlier tokens.
The success of GPT-3 validated the scalability and flexibility of the decoder-only architecture at ultra-large scale, after which mainstream research converged toward decoder-only.
All Facts (3)
The decoder-only architecture uses only the Transformer decoder stack, predicting the next token one by one in an autoregressive manner through a causal attention mask, learning during training to predict subsequent content based on preceding context.
50%UnverifiedIn a decoder-only architecture, the causal mask ensures each token can only see itself and earlier tokens.
50%UnverifiedThe success of GPT-3 validated the scalability and flexibility of the decoder-only architecture at ultra-large scale, after which mainstream research converged toward decoder-only.
50%