DinoGPT
An open-source miniature character-level Transformer model with approximately 53,000 parameters, implemented from scratch by hand, designed specifically to generate dinosaur names, and accompanied by an interactive Streamlit demo.
Timeline (last 90 days)
A developer implemented a tiny Transformer called DinoGPT from scratch in PyTorch, specifically for generating dinosaur names.
DinoGPT has only about 53,000 parameters.
DinoGPT's training data is 1,535 real dinosaur names.
DinoGPT comes with a Streamlit web app where users can enter the first few letters to generate dinosaur names.
DinoGPT's attention mechanism is hand-written by the author, without directly calling ready-made high-level APIs.
DinoGPT uses character-level modeling, with a vocabulary of 26 English letters plus a few punctuation marks.
Tiny models like DinoGPT typically use learnable positional embeddings, with the parameter count equal to the maximum sequence length multiplied by the hidden dimension.
The DinoGPT project covers the complete pipeline of data processing, model implementation, training and tuning, and productized deployment.
In the post, the author invites feedback, including whether the interaction is clear, how to make it more fun, the most outrageous dinosaur names, and any errors in the code implementation or explanation.
All Facts (9)
A developer implemented a tiny Transformer called DinoGPT from scratch in PyTorch, specifically for generating dinosaur names.
50%UnverifiedDinoGPT has only about 53,000 parameters.
50%UnverifiedDinoGPT's training data is 1,535 real dinosaur names.
50%UnverifiedDinoGPT comes with a Streamlit web app where users can enter the first few letters to generate dinosaur names.
50%UnverifiedDinoGPT's attention mechanism is hand-written by the author, without directly calling ready-made high-level APIs.
50%UnverifiedDinoGPT uses character-level modeling, with a vocabulary of 26 English letters plus a few punctuation marks.
50%UnverifiedTiny models like DinoGPT typically use learnable positional embeddings, with the parameter count equal to the maximum sequence length multiplied by the hidden dimension.
50%UnverifiedThe DinoGPT project covers the complete pipeline of data processing, model implementation, training and tuning, and productized deployment.
50%UnverifiedIn the post, the author invites feedback, including whether the interaction is clear, how to make it more fun, the most outrageous dinosaur names, and any errors in the code implementation or explanation.
50%