WordPiece
A subword tokenization algorithm that determines symbol merges by maximizing language model likelihood. It serves as the core tokenization method adopted by models such as BERT, producing a vocabulary of subword units that balances coverage and compactness.
Core Facts
Modern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
Different models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.
Timeline (last 90 days)
Porting LLaVA-1.5's token IDs as-is to two models with disjoint vocabularies drops F1 straight to 0.00, whereas using tokenizer-derived readings keeps F1 stable between 0.80 and 0.92.
The researchers conducted a four-reading audit on another four VLMs covering three tokenizer families, concluding that the Yes/No reading confusion is a protocol-level rather than model-level issue.
Subword tokenizers (such as BPE, WordPiece) split identifiers like E4012 into common fragments such as E, 40, 12, diluting exact character information and causing identifiers to lose discriminability in embedding space.
Different models have different tokenizers, such as BPE and WordPiece.
Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
In a warm English encoding with bert-base-uncased, Normalization consumed 88% of the encoding time, while WordPiece accounted for only 8%.
BAAI/bge-small-en-v1.5 uses a WordPiece tokenizer, which splits snake_case and camelCase into meaningless fragments.
The WordPiece tokenizer's ## prefix marks the latter part of a subword; for example, gatito is split into ga, ##ti, ##to, and gato also contains ##to, causing shared tokens between the query and the document and producing false similarity.
A Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
The choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
5 more timeline events
All Facts (16)
Modern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
80%VerifiedSubword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
65%VerifiedDifferent models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.
65%UnverifiedThe researchers conducted a four-reading audit on another four VLMs covering three tokenizer families, concluding that the Yes/No reading confusion is a protocol-level rather than model-level issue.
50%UnverifiedPorting LLaVA-1.5's token IDs as-is to two models with disjoint vocabularies drops F1 straight to 0.00, whereas using tokenizer-derived readings keeps F1 stable between 0.80 and 0.92.
50%UnverifiedSubword tokenizers (such as BPE, WordPiece) split identifiers like E4012 into common fragments such as E, 40, 12, diluting exact character information and causing identifiers to lose discriminability in embedding space.
50%UnverifiedDifferent models have different tokenizers, such as BPE and WordPiece.
50%UnverifiedIn a warm English encoding with bert-base-uncased, Normalization consumed 88% of the encoding time, while WordPiece accounted for only 8%.
50%UnverifiedBAAI/bge-small-en-v1.5 uses a WordPiece tokenizer, which splits snake_case and camelCase into meaningless fragments.
50%UnverifiedThe WordPiece tokenizer's ## prefix marks the latter part of a subword; for example, gatito is split into ga, ##ti, ##to, and gato also contains ##to, causing shared tokens between the query and the document and producing false similarity.
50%UnverifiedA Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.
50%UnverifiedWordPiece was proposed by Google and adopted by BERT; its merge criterion maximizes the likelihood of a language model rather than purely counting frequencies.
50%UnverifiedThe choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
50%UnverifiedWordPiece determines its merging strategy with the objective of maximizing language model likelihood.
50%UnverifiedMainstream subword tokenizers (such as BPE and WordPiece) treat different written forms of the same word (e.g., hello, Hello, HELLO, Héllo) as independent, mutually unrelated vocabulary entries.
50%UnverifiedFixed vocabulary sizes typically range from 30,000 to 100,000 entries.
50%