Selective Recurrence
A model architecture technique that recursively applies a small number of early Transformer layers multiple times at each step of the denoising process, enabling deeper computation under an equal FLOP budget and decoupled from the training objective.
Timeline (last 90 days)
Recurrent UDLM recursively reuses early Transformer layers, improving training efficiency several-fold while keeping the training objective and sampler unchanged.
Selective recurrence was previously shown in MDLM to improve training efficiency under an equal FLOP budget, specifically by recursively applying a few early Transformer layers at each denoising step.
Selective recurrence is a purely architectural modification decoupled from the training objective and can be applied independently without changing the diffusion model's training objective or sampling algorithm.
All Facts (3)
Recurrent UDLM recursively reuses early Transformer layers, improving training efficiency several-fold while keeping the training objective and sampler unchanged.
50%UnverifiedSelective recurrence was previously shown in MDLM to improve training efficiency under an equal FLOP budget, specifically by recursively applying a few early Transformer layers at each denoising step.
50%UnverifiedSelective recurrence is a purely architectural modification decoupled from the training objective and can be applied independently without changing the diffusion model's training objective or sampling algorithm.
50%