Unverified50% confidenceFactExact time
BPE was introduced into the field of natural language processing for subword tokenization by Sennrich et al. in 2016.
1
Sources
50%
Confidence
Long-term
Relevance
9/3/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedMainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.70% similarUnverified现代语言模型的词汇表规模通常在32,000到100,000个token之间，使用BPE或SentencePiece等分词方法64% similarUnverified研究界逐步转向结合大语言模型的新一代评估框架如COMET和BERTScore以更好捕捉语义层面翻译质量61% similarUnverifiedThe tiktoken tokenizer used by OpenAI is based on a BPE variant, where common English words are typically preserved as whole units while rare words are split into multiple subword units.61% similarUnverifiedThe probing technique was first widely applied in the field of natural language processing to detect whether language models encode linguistic features such as syntactic structures and semantic roles, with Alain and Bengio doing pioneering work in 2016.60% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/849953API
curl https://kongchang.com/api/v1/knowledge/claims/849953MCP
get_claim(id=849953)