Unverified50% confidenceFactExact time
Gemma系列使用SentencePiece分词器，在BPE基础上引入Unigram语言模型，词表约含256,000个条目
1
Sources
50%
Confidence
Long-term
Relevance
7/15/2026
First Seen
Sources
GDScript+Vulkan在Godot中运行Gemma大语言模型实验
Related Claims
UnverifiedThe SentencePiece tokenizer used by the Gemma series has a vocabulary of approximately 256,000 entries.85% similarVerifiedMainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.68% similarUnverifiedWhisper uses a Byte Pair Encoding (BPE) tokenizer with a multilingual vocabulary of approximately 50,000 tokens, supporting 99 languages.65% similarUnverified为英语优化的BPE分词器会将印度语言文本切分为过多子词单元，导致序列长度膨胀和信息密度下降62% similarUnverifiedBenchmark datasets for sign language translation (such as PHOENIX-2014T, CSL-Daily, and How2Sign) typically contain only thousands to tens of thousands of samples, far smaller than the millions of sentence pairs in parallel corpora in the machine translation field.60% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/517765API
curl https://kongchang.com/api/v1/knowledge/claims/517765MCP
get_claim(id=517765)