Unverified50% confidenceFactExact time
Subword tokenizers (such as BPE, WordPiece) split identifiers like E4012 into common fragments such as E, 40, 12, diluting exact character information and causing identifiers to lose discriminability in embedding space.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
Unverified大多数主流分词器（BPE、WordPiece）将数字按字符级别分割，导致数字的位值结构在输入阶段就已被破坏77% similarUnverifiedSubword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.72% similarUnverifiedEmbedding models encode the topic of a passage rather than exact characters, causing adjacent error codes (such as E4012 and E4021) to nearly coincide in vector space, with ranking being nearly random.69% similarUnverifiedA grapheme cluster is the smallest unit of writing as perceived by humans and may be composed of multiple scalar values, for example e plus a combining acute accent, or a family emoji assembled from multiple code points joined by zero-width joiners.66% similarUnverifiedSparse retrieval methods based on term frequency, such as BM25, will miss a large number of recalls when the same concept is expressed in multiple different ways.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1072977API
curl https://kongchang.com/api/v1/knowledge/claims/1072977MCP
get_claim(id=1072977)