CopyLoop: A Training-Free Decoding Acceleration Method for Looped Transformers

Title

CopyLoop: A Training-Free Decoding Acceleration Method for Looped Transformers

Meta Description

Looped Transformer decoding is slow? CopyLoop introduces a training-free context-copy proposal and shallow-layer screening method, achieving up to 10.70x decoding speedup across multiple model families, with strong performance on code generation and math reasoning tasks.

Article Content

Looped Transformers boost computational capacity by repeatedly applying a shared block, but the loop depth makes conventional full-depth decoding expensive. Existing speculative decoding pipelines can accelerate generation, yet they are constrained by the overhead of sequential decoding. A new paper proposes the CopyLoop algorithm, which achieves training-free decoding acceleration by combining low-cost context-copy proposals, shallow-layer screening, and continuous loop scheduling.

The Decoding Dilemma of Looped Transformers and CopyLoop's Breakthrough Approach

The core advantage of looped Transformers lies in increasing the model's "thinking depth" by repeatedly invoking the same set of parameter blocks. This design allows the model to perform more complex computation during inference, making it especially suited to tasks that require multi-step reasoning. But there is a cost behind the advantage: for every token generated, the model must run through the entire loop depth, making decoding speed a bottleneck.

Existing acceleration schemes, such as speculative decoding, typically have a small model quickly generate candidate tokens, which are then verified by a large model. But this approach has limited effectiveness on looped Transformers, because the verification process itself also requires full-depth computation, and there is a sequential dependency between candidate generation and verification.

CopyLoop's entry point is clever: instead of generating tokens from scratch, it extracts candidate suffixes from the repeated context. Specifically, the algorithm uses fragments already present in the input as "proposals," because in many generation tasks (such as code completion and math reasoning), subsequent content often repeats or resembles structures in the preceding context. The cost of such "copy" proposals is extremely low, requiring almost no additional computation.

Speculative Decoding is a technical framework for accelerating autoregressive generation: it first uses a smaller, faster "draft model" to generate multiple candidate tokens at once (called drafts), and then the original large model verifies these candidates in parallel, accepting the token prefix that is consistent with its own distribution. Because verification can be done in parallel, overall generation speed improves while the output distribution remains consistent with the original model. On standard Transformers, the draft model and the target model are independent of each other, and the verification process requires only a single forward pass. But in looped Transformers, verification of each token requires running through the full loop depth, and there is a strict sequential dependency between draft generation and verification, which greatly weakens the parallel advantage of speculative decoding. This is precisely the bottleneck that CopyLoop attempts to break through.

Shallow-Layer Screening and Continuous Scheduling: The Two Pillars of CopyLoop

CopyLoop's workflow is divided into two stages: proposal and screening. First, it copies multiple candidate suffixes from the context, and these suffixes may contain multiple tokens. Then, rather than immediately performing full-depth verification, it quickly evaluates these candidates at shallow loop depths. Shallow evaluation requires far less computation than full depth, so it can efficiently filter out obviously incorrect candidates.

More critically, there is the continuous loop scheduling mechanism. In the traditional process, verification must wait for all blocks to complete full-depth computation. CopyLoop, however, allows newer blocks to be processed concurrently while earlier blocks are still advancing toward full depth. This pipeline-style scheduling overlaps computations at different depths, further compressing overall latency.

The paper's ablation experiments confirm a strong synergistic effect between shallow-layer screening and continuous scheduling—using either technique alone is significantly less effective than combining both.

The core idea of shallow-layer screening is to exploit the "anytime" property of looped Transformers: at shallower loop depths, the model can already produce a preliminary token distribution. Although its accuracy is lower than at full depth, the computation is proportionally reduced. CopyLoop scores the copied candidate suffixes at low depth, quickly eliminates low-probability candidates, and retains only a few promising candidates for full-depth verification. The benefit of this approach is that expensive full-depth computation is concentrated on the most valuable candidates, avoiding wasted compute on obviously incorrect proposals. The screening threshold (i.e., the minimum score for accepting a candidate into full-depth verification) needs to be determined through lightweight calibration to balance recall and computational overhead.

Continuous Loop Scheduling draws on the idea of pipeline parallelism: in traditional decoding, the t-th token cannot begin computation until all loop blocks for the (t-1)-th token have completed, forming a serial dependency. CopyLoop treats loop depth as overlappable pipeline stages—while earlier blocks are still advancing toward deeper loops, subsequent blocks can already begin proposing and screening based on shallow results. Computations at different depths overlap in time, hardware utilization improves, and end-to-end latency decreases accordingly. This scheduling requires the algorithm to manage the states of multiple blocks at different depths and to work closely with shallow-layer screening: the earlier shallow results become available, the earlier the pipeline can start processing the next block.

Experimental Validation: Significant Speedups Across Model Families

The research team evaluated CopyLoop on six checkpoints from three model families: Ouro, Raven, and Huginn, with tasks covering code generation and multi-step math reasoning.

The results show that on Base models, CopyLoop achieved decoding speedups ranging from 2.98x to 10.70x compared with full-depth autoregressive decoding; on Thinking models, the speedup ranged from 2.74x to 3.52x. Across all evaluated tasks, CopyLoop either outperformed all competing baselines or performed very close to them.

Interestingly, CopyLoop requires only lightweight model calibration and no retraining. Moreover, analysis shows that the speedup is more pronounced in scenarios with longer generation lengths and reusable context. This is especially beneficial for scenarios such as code generation that naturally contain repetitive patterns.

Implications and Insights for the Deployment of Looped Transformers

Looped Transformers are regarded as an important direction for improving model reasoning capabilities, but decoding efficiency has always been an obstacle to their deployment. The significance of CopyLoop is that it provides a training-free and easy-to-integrate acceleration scheme. This means existing looped Transformer models can gain speed improvements at almost zero cost.

From a broader perspective, CopyLoop's approach—using context copying as proposals, shallow-layer screening as filtering, and continuous scheduling as a pipeline—may inspire decoding optimization for other model architectures that require multi-step computation. Especially in scenarios requiring long-sequence generation, this strategy of "reusing existing context" is worth further exploration.

Of course, CopyLoop also has its applicable boundaries. Its speedup depends on the existence of reusable fragments in the context; for fully open-domain generation tasks, the proposal hit rate may decline. In addition, the threshold setting for shallow-layer screening may need to be adjusted for different models. The lightweight calibration mentioned in the paper is precisely intended to address this issue.

Overall, CopyLoop provides a practical and effective solution for efficient decoding of looped Transformers, and it also opens a new path for applying speculative decoding to looped architectures.

One-line Summary

CopyLoop uses context-copy proposals, shallow-layer screening, and continuous scheduling to achieve training-free decoding acceleration for looped Transformers.

Paragraph Summary

Looped Transformers increase computational depth by repeatedly applying shared blocks, but full-depth decoding is expensive, and existing speculative decoding is limited by sequential dependencies. CopyLoop proposes a training-free acceleration scheme: copy candidate suffixes from context as low-cost proposals, quickly screen them at shallow loop depths, and overlap computations at different depths through continuous loop scheduling in a pipeline manner. On six checkpoints from three model families—Ouro, Raven, and Huginn—Base models achieve 2.98x to 10.70x speedups, and Thinking models achieve 2.74x to 3.52x speedups, requiring only lightweight calibration. Ablation experiments show that shallow-layer screening and continuous scheduling have a strong synergistic effect, and the speedup is especially significant in scenarios with long generation and reusable context, such as code generation.