Character-Level Modeling
A granularity of language modeling that uses individual characters as the smallest input unit, requiring no tokenizer and a very small vocabulary. It is well suited to observing how a model learns spelling regularities, but produces longer sequences.
Timeline (last 90 days)
DinoGPT uses character-level modeling, with a vocabulary of 26 English letters plus a few punctuation marks.
Character-level modeling differs from the BPE subword tokenization commonly used by the GPT series: subword tokenization can compress sequence length and capture common word fragments, but requires training an additional tokenizer.
All Facts (2)
DinoGPT uses character-level modeling, with a vocabulary of 26 English letters plus a few punctuation marks.
50%UnverifiedCharacter-level modeling differs from the BPE subword tokenization commonly used by the GPT series: subword tokenization can compress sequence length and capture common word fragments, but requires training an additional tokenizer.
50%