Building a Transformer from Scratch: A 53K-Parameter Dinosaur Name Generator

A developer implemented a tiny Transformer from scratch in PyTorch, named DinoGPT, specifically designed to generate dinosaur names. The project has only about 53,000 parameters, is trained on 1,535 real dinosaur names, and comes with a Streamlit web app that lets anyone type in a few starting letters and see what dinosaur gets "hatched."

What the project is: a character-level name generator

DinoGPT works in a straightforward way: you give it the beginning of a name—your own name, a nickname, or just a few random letters—and it predicts the rest letter by letter, assembling a complete dinosaur name. The model is small enough to run easily in a browser, but it's fully featured despite its size—the attention mechanism was hand-written line by line by the author.

The author is refreshingly candid in the post: the core purpose of building this wasn't to chase generation quality, but to understand how Transformers actually work. This "reverse-learn through a minimal viable project" approach is more effective for many developers wanting to get into AI than reading ten papers.

Character-level Modeling means the model uses individual characters rather than words as the smallest input unit. In DinoGPT, the vocabulary is just the 26 English letters plus a few punctuation marks, so the embedding matrix and output layer dimensions are extremely small. This differs from the BPE (Byte Pair Encoding) subword tokenization commonly used in the GPT series: subword tokenization compresses sequence length and captures common word fragments, but requires training a separate tokenizer; the character-level approach eliminates the tokenization step entirely, resulting in longer sequences but finer granularity at each step, which is ideal for observing how the model learns spelling patterns from scratch. The trade-off is longer sequences and harder long-range dependencies, but for short-text tasks like dinosaur names, this is barely an issue.

Why 53k parameters are worth paying attention to

In an era of hundred-billion-parameter large models, 53,000 parameters looks like a toy. But that's precisely what makes this project most valuable: it compresses the complete Transformer training and inference pipeline into a codebase you can read end to end.

Character-level modeling means a tiny vocabulary (26 letters plus a few symbols), and the embedding layer, positional encoding, multi-head attention, and feed-forward network are all present—but each layer's dimensions are small enough to work through by hand. For anyone wanting to understand "what attention is actually attending to," this scale is an advantage—you can genuinely track how each weight changes.

The training data is only 1,535 dinosaur names, which is an interesting scale in itself. Less data means higher overfitting risk, but the author's goal is demonstration rather than production, so what the model learns is the spelling "style" of dinosaur names—common Latinized roots, diphthong combinations, the -saurus or -raptor ending patterns—rather than real paleontological taxonomic knowledge.

Positional Encoding is a key component in Transformers for injecting order information. Self-attention itself is permutation-invariant—shuffle the input order and the attention output simply reorders accordingly; the model can't tell which letter comes first. There are two common approaches: fixed sinusoidal/cosine positional encoding that isn't trained, or learnable positional embeddings optimized as parameters alongside the model. Tiny models like DinoGPT typically use learnable positional embeddings, with parameter count equal to max sequence length times hidden dimension. Understanding positional encoding helps clarify a key fact: the Transformer's "sense of order" isn't innate—it's explicitly added, and this is one of the most fundamental structural differences from RNNs.

From code to playable: Streamlit lowers the barrier to experience

The author packaged the model into a Streamlit app, which is the key step that turns the entire project from a "personal experiment" into a "shareable work." Streamlit's advantage is that you barely need to write any frontend code—a few dozen lines of Python can build an interactive page with an input box and output results.

From the demo screenshot, the interface is designed around the single main flow of "input a start, generate a name," with no extraneous buttons or configuration options. The first question the author asks in the post is "is it clear what to input and what the app does," showing he cares about first-time users' comprehension cost rather than showing off. This kind of product awareness is rare in personal projects.

Streamlit is a Python framework for data science and machine learning. Developers just write ordinary scripts, declaring interface elements with functions like st.text_input and st.button, and the framework handles frontend rendering and state refresh automatically. Its execution model is "the script reruns top to bottom": every user interaction re-executes the entire script, so model loading typically needs to be cached with @st.cache_resource to avoid reinitializing weights on every click. For a tiny model like DinoGPT with extremely fast inference, this pattern has almost no performance overhead; its greatest value lies in compressing the distance from "model → shareable web page" to a few dozen lines of code, letting personal experiments quickly become works others can experience.

What kind of feedback the author wants

The questions listed in the post are very specific: is the interaction clear, how to make it more fun, has anyone generated the most absurd dinosaur name, and corrections to code implementation or explanations are welcome. The last one is especially noteworthy—he's publishing the project as learning notes and actively inviting those in the know to point out mistakes, which makes for more valuable discussion than posts that simply showcase results.

For those wanting to reproduce similar projects, the build notes on GitHub may be more useful than the code itself. The pitfalls encountered when hand-writing the attention layer from scratch, the debugging process when dimensions don't match, the design trade-offs of the character-level tokenizer—these details are often invisible in tutorials.

Big lessons from a small project

The significance of projects like DinoGPT isn't in how realistic the generated dinosaur names are, but in demonstrating a low-cost AI learning path: pick an interesting small task, run the complete pipeline through with the smallest model scale, then turn the result into something others can play with. The whole process covers data processing, model implementation, training and tuning, and product deployment—each step can be explored independently.

If you also want to understand Transformers, rather than starting from deriving formulas, it's better to set a small goal like "generate some kind of name" and just get the model running first. 53k parameters is enough to let you see the input and output of each layer clearly, and a demo you can share adds an extra layer of external feedback to drive your learning.

One-line Summary

Developer hand-writes a 53K-parameter Transformer in PyTorch to generate dinosaur names and turns it into a playable demo.

Paragraph Summary

DinoGPT is a character-level Transformer implemented from scratch by a developer, with only about 53,000 parameters, trained on 1,535 real dinosaur names. Users input the beginning of a name, and the model predicts letter by letter to assemble a new dinosaur name. The project is packaged as a Streamlit web app, shareable with almost no frontend code. The author candidly states the goal isn't generation quality but understanding Transformer internals through a minimal viable project: the attention layer hand-written line by line, with components small enough to track how each weight changes. The article argues that this path of "small task + minimal model + shareable demo" covers data processing, model implementation, training and tuning, and product deployment, making it a low-cost and efficient AI learning route that builds intuition better than simply reading papers.