#Transformer
28 related articles
Understanding AI Large Models in Ten Minutes: Tokens, Vectors, and the Transformer Principle
From tokens and vector embeddings to the Transformer self-attention mechanism, this systematically breaks down the three stages of AI large model pretraining, fine-tuning, and inference, explaining why large models output word by word, where hallucinations come from, and how they understand contextual semantics.
Calculating the Transformer Forward Pass by Hand: Understanding Large Models Matrix by Matrix
A Reddit user shares a method to understand large model internals by hand-calculating every matrix in the Transformer forward pass, covering positional encoding, encoder shape invariance, and BERT's relation to the encoder, with step-by-step practice resources.
New Insights into Transformer Generalization Boundaries: Why Long-Sequence Training Is More Efficient Than More Samples
New research jointly analyzes in-distribution and length generalization for Transformers, finding that increasing training length is exponentially more efficient than adding samples, providing theoretical support for long-sequence training strategies.
Generative AI for Developers: FlashAttention, GQA, and MoE Architecture Explained
Explaining Lesson 2 of Generative AI For Devs: how FlashAttention reduces GPU memory transfers, how GQA and MQA compress the KV cache by sharing key-value heads, how Mixture of Experts scales model capacity with fewer activated parameters, and the performance-quality trade-offs of each.
Building a Transformer from Scratch: A 53K-Parameter Dinosaur Name Generator
A developer hand-builds a 53K-parameter Transformer called DinoGPT in PyTorch, trained on 1,535 dinosaur names and wrapped in a Streamlit demo. We unpack the technical approach, the value of tiny models for learning, and product lessons.
Implementing the Transformer Architecture from Scratch: Training Acceleration and Performance Improvement in Practice
A Reddit developer implemented the Transformer from scratch but training was slow and results poor. This article analyzes bottlenecks and offers PyTorch 2.0 attention optimization, mixed precision, learning rate scheduling, and small-data improvement strategies.
The Glass Transformer: Peeking Inside the Model with an 11-Minute Training Run
Reddit developer Rick shares "The Glass Transformer": an 11-minute free Kaggle training run that traces token flow from input to output, generating 1,000+ pages of tracing data—a low-cost diagnostic for Transformer alternative architectures.
Asymmetric Loop Transformer: A New Paradigm with Fixed Global Representations and Iterative Decoding
Asymmetric Loop Transformer (AsymLoop) separates single-pass context encoding from recurrent state refinement and fixes the global KV cache, raising average commonsense reasoning accuracy from 49.58% to 50.85% at 1B scale, offering a new design paradigm for looped language models.
How an LLM Generates a Word: The Complete Pipeline from Token to KV Cache
From tokenization to transformer attention, from Prefill/Decode phases to KV Cache costs, and why LLMs have no memory—a clear explanation of the complete pipeline for generating a word and the root cause of expensive long context.
Deriving the Transformer from Scratch: You Could Have Invented It from Memory
From RNN bottlenecks to attention, this article breaks down the logic of deriving the Transformer and shows why you could reinvent the architecture from memory with reproducible building blocks.
Cross-Market Meta-Learning: How MetaGrid Improves Electricity Price Forecasting Accuracy
MetaGrid uses cross-market meta-learning and a topology-aware Transformer to achieve the lowest MAE on electricity price forecasting across five U.S. ISOs, improving 3.1% over the best baseline and 7.7% over training from scratch.
RoPE Decay Curve Is Noise: Taking Absolute Values Won't Fix It
The RoPE attention decay curve is actually sampling noise, and taking absolute values won't fix it. This article explains the relativity proof of rotary position encoding, the nature of phase dephasing, an LCG seed-related bug, and why NTK scaling cannot act on the fastest frequency pair, with 226 verification assertions and measured data.
Federated Learning + Temporal Transformer: A New Framework for CAN Bus Anomaly Detection in Vehicles
A new arXiv paper proposes a CAN bus anomaly detection framework combining a Temporal Transformer encoder with Federated Lightweight Heads, enabling privacy-preserving, efficient anomaly detection for connected vehicles.
How Softmax Attention Learns Decision Thresholds: A Theoretical Study of Training Dynamics
A theoretical study on how a two-parameter softmax attention model learns decision thresholds via gradient pretraining — covering error decomposition, coordinated parameter divergence, and single-head limitations.
AttSVD: Attention-Guided SVD Compresses KV Cache to Half the Memory Without Accuracy Loss
AttSVD compresses KV cache via attention-guided SVD along the feature axis — no training required, 50% memory, full accuracy on LongBench and agentic benchmarks.
Transformer, RNN, SSM: Where Does an AI Model's Memory Actually Live?
A fresh perspective comparing RNN recurrent states, Transformer KV caches, and SSM/Mamba selective states through the lens of "where memory lives" in AI models.
Yandex Sona: How a Single Transformer Replaced 15+ Recommendation Components
Yandex Music's Sona uses a single Transformer to replace 15+ recommender components. A/B tests show +4.53% active users and +6.30% listening time, with History Compression cutting inference costs in half.
TurboGPT: A Minimalist Experiment in Training a 22KiB Transformer in 13 Seconds
TurboGPT is a minimalist AI project on Hacker News that trains a 22KiB Transformer in just 13 seconds. We explore its value and limits in education, reproducibility, and edge AI.
Understanding LLM Inference: Why Generation Is Slow and How to Speed It Up
A deep dive into why LLM inference is memory-limited, covering KV cache, GQA, MLA, SSMs, diffusion models, quantization, pruning, and speculative decoding — based on Stanford's LLM course.
Visualizing the Transformer: Breaking Down the Core Architecture Behind Large Language Models
A visual breakdown of the Transformer architecture — covering self-attention, multi-head attention, positional encoding, and encoder-decoder design to help AI learners build intuition for LLMs.