Unverified50% confidenceFactExact time

mBERT achieves a fertility of 1.10 tokens/word on Odia, but at the same time discards 87.9% of the text as UNK.

1

Sources

50%

Confidence

Long-term

Relevance

10/20/2026

First Seen

Sources

Tokeniser Coverage: The Invisible Ceiling of Small Language Models in Multilingual Scenarios

rss10/17/2026

Related Entities

mBERTmodel 奥里亚语concept

Related Claims

UnverifiedThe same 1,000-character Chinese text produces approximately 800 tokens under GPT-4o's tiktoken (cl100k_base), approximately 750 under the Claude tokenizer, and approximately 900 under Llama's sentencepiece tokenizer.61% similar

Cite This Claim

Stable URI

https://kongchang.com/claim/1074689

API

curl https://kongchang.com/api/v1/knowledge/claims/1074689

MCP

get_claim(id=1074689)