GAP Architecture: A Transformer Experiment Merging Attention and MLP

The standard Transformer structure—multi-head attention followed by a feed-forward network (FFN)—has become almost the default recipe over the past few years. But some researchers are beginning to try more radical changes: merging these two sublayers into a single gated module. An experimental architecture called GAP (Gated Attention Projection), shared by a developer on Reddit, is an exploration in exactly this direction.

GAP's Core Idea: One Gated Block Replacing Two Sublayers

GAP's basic proposition is straightforward: instead of stacking attention layers and MLP layers independently, it fuses attention computation and MLP-style projection processing into the same gated block. The gating mechanism here takes on part of the regulatory role that residual connections and normalization between the two layers implicitly handled in the standard structure.

According to the author, the motivation behind this design is parameter efficiency—in a standard Transformer, attention and FFN each account for a substantial proportion of parameters, and if a single gated structure with shared projections could cover both types of computation, there would be an opportunity to compress the total parameter count. It's worth emphasizing that the author explicitly states in the post: this is an experimental architecture, and there is no claim that it is overall superior to the Transformer.

Gating in neural networks typically refers to using a learnable scalar or vector to perform element-wise multiplicative modulation on another signal, written as (y = g \odot x), where the gate value (g) is computed from the input via functions such as sigmoid or softmax. Its role is to let the network dynamically decide the proportion of information that passes through, thereby achieving a kind of "soft switch" selectivity. Gating has long been used in LSTM's forget gate and GRU's update gate, and in recent years it has also been introduced into Transformer variants—for example, gated linear units (GLU) and their derivatives replace some activation functions in the FFN. GAP places gating at the point where attention and projection are fused, with the idea that the same multiplicative modulation serves both types of computation, rather than relying on residual connections and LayerNorm between two layers for implicit regulation as in the standard structure.

Experimental Results: Fewer Parameters, But No Accuracy Win

The author conducted a controlled experiment on WikiText-103, with each model using 100 million scored training tokens. The main findings can be viewed at several levels:

The parameter compression is substantial. A smaller configuration can reduce the parameter count by 34.31%, but this version's performance dropped noticeably relative to the standard Transformer, so the author did not adopt it. The main model tested, GAP20, gives back about 14% of the saved parameters to the model to increase width, with a net effect of 19.91% fewer parameters than the standard Transformer—which is also the origin of the name "GAP20."

The standard Transformer is better on language model accuracy. This is the result the author honestly reports, without avoiding it. GAP20 lags behind the control group in final language model accuracy, indicating that the improvement in parameter efficiency did not directly translate into an advantage in modeling capability.

Training throughput has a bright spot. In CPU benchmarks, after compiling GAP20 with torch.compile, training throughput improved by about 19.9% compared to the uncompiled version. This number only reflects the speedup brought by compilation, not GAP's speed advantage over the standard Transformer, and this distinction needs to be kept clear when interpreting it.

WikiText-103 is a commonly used medium-scale corpus in the language modeling field, derived from Wikipedia and at roughly the 100-million-word level, often used to evaluate a model's ability to model long-range dependencies. The so-called "scored training tokens" refers to the number of tokens involved in loss computation, which is usually not equal to the total word count of the corpus, because some tokens may be masked or used for special processing. Using 100 million scored tokens as a unified budget for comparison is a common practice to control training compute, making it convenient to compare different architectures under the same amount of training. It should be noted that experiments at this scale are medium-scale validation, and conclusions are quite sensitive to hyperparameters, learning rate schedules, and the model's width-to-depth ratio, so the results of a single comparison are better treated as directional reference rather than a definitive conclusion.

torch.compile is the compilation frontend introduced in PyTorch 2.0, which improves training and inference speed through optimizations such as operator fusion and kernel generation on the computation graph. The speedup it brings comes from the compiler's rewriting of the existing computation graph and has nothing to do with the model architecture's own design; therefore, the roughly 19.9% throughput improvement in the article should be understood as "GAP20 is faster after compilation than its uncompiled self," not as a speed advantage of GAP over the standard Transformer. When comparing speed across architectures, the compilation state, hardware, and batch size must be kept consistent, otherwise the numbers cannot be directly compared.

Architecture Code and Configurable Sizes

The author has released the architecture code, configurable model sizes, speed benchmarks, and a research paper in the GitHub repository (rickey1990/Gated-Attention-Projection). For those who want to reproduce or run further experiments, this set of materials is fairly complete—it includes both a runnable implementation and a written explanation of the motivation.

From an engineering perspective, a potential appeal of the gated fusion structure is its friendliness to compilers and hardware: the computation path within a single block is more concentrated, offering more fusion opportunities, which may be interrelated with the throughput improvement brought by torch.compile. However, this is only speculation; the author did not argue for this causal relationship in the post.

What This Experiment Shows

With about 20% fewer parameters and a slight drop in accuracy, whether this trade-off is worthwhile depends on how sensitive the deployment scenario is to model size and inference cost. In edge devices or scenarios requiring strict control of VRAM, the approach of structures like GAP has reference value; but in training tasks pursuing benchmark accuracy, the standard Transformer remains the safer baseline.

The author openly solicits feedback on the design, mathematical derivation, and directions for improvement, and this openness itself deserves recognition. Architectural innovation rarely gets it right the first time, and GAP currently looks more like a hypothesis test with clear data support than a final solution.

What do you think of the design that fuses attention and MLP? If your scenario is sensitive to parameter count, would you consider trying such a structure?

One-line Summary

Experimental architecture GAP merges attention and MLP into a gated block, saving about 20% of parameters but slightly reducing accuracy.

Paragraph Summary

The article introduces an experimental architecture called GAP (Gated Attention Projection) proposed by a Reddit developer. Its core idea is to fuse the Transformer's separate multi-head attention sublayer and feed-forward network sublayer into a single gated block, using shared projections to cover both types of computation, with the goal of compressing the parameter count. The author ran a comparison on WikiText-103 with 100 million scored tokens: the main model GAP20 has 19.91% fewer parameters than the standard Transformer, but its final language model accuracy lags behind; a more aggressive configuration can reduce parameters by 34.31%, but performance drops noticeably. torch.compile improves GAP20's training throughput by about 19.9% on CPU benchmarks, but this only reflects compilation speedup. The author explicitly positions it as experimental exploration, has released the code, configurable sizes, speed benchmarks, and paper, and does not claim it is overall superior to the Transformer.