Unverified60% confidenceFactExact time
SentencePiece is a language-agnostic tokenization framework proposed by Google that models directly on raw text and is suitable for multilingual use and languages without clear word boundaries.
2
Sources
60%
Confidence
Long-term
Relevance
8/28/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSentencePiece由Google的Kudo和Richardson在2018年提出，将输入文本视为原始字符流，不依赖预分词步骤76% similarUnverifiedWordPiece was proposed by Google and adopted by BERT; its merge criterion maximizes the likelihood of a language model rather than purely counting frequencies.76% similarUnverifiedThe SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.72% similarVerifiedSubword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.69% similarUnverifiedOpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/813374API
curl https://kongchang.com/api/v1/knowledge/claims/813374MCP
get_claim(id=813374)