BPE
A mainstream LLM tokenization algorithm originating from data compression. It iteratively merges high-frequency adjacent character pairs to build a subword vocabulary, naturally handling out-of-vocabulary words.
Core Facts
The BPE algorithm originates from the field of data compression and builds a vocabulary by iteratively merging high-frequency character pairs.
Mainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.
Modern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
Timeline (last 90 days)
Vocabulary design (such as the granularity of BPE merging) not only affects computational efficiency but also indirectly affects the difficulty of model generalization.
Mainstream large models rely on tokenizers (such as the BPE algorithm) to split text into subword units, with each token corresponding to an ID in the vocabulary, which typically contains tens of thousands to hundreds of thousands of entries.
Character-level modeling differs from the BPE subword tokenization commonly used by the GPT series: subword tokenization can compress sequence length and capture common word fragments, but requires training an additional tokenizer.
llama.cpp's tokenizer implementation follows the two mainstream schemes of SentencePiece and BPE, and provides a unified abstraction for the vocabulary embedded in GGUF, covering details such as special tokens, byte fallback and normalization rules.
After BPE tokenization, Chinese is roughly one token per character, so 512 tokens is approximately equal to 512 Chinese characters.
Porting LLaVA-1.5's token IDs as-is to two models with disjoint vocabularies drops F1 straight to 0.00, whereas using tokenizer-derived readings keeps F1 stable between 0.80 and 0.92.
The researchers conducted a four-reading audit on another four VLMs covering three tokenizer families, concluding that the Yes/No reading confusion is a protocol-level rather than model-level issue.
Subword tokenizers (such as BPE, WordPiece) split identifiers like E4012 into common fragments such as E, 40, 12, diluting exact character information and causing identifiers to lose discriminability in embedding space.
Different models have different tokenizers, such as BPE and WordPiece.
The BPE training process iteratively counts the frequencies of adjacent character pairs and merges the most frequent pair into a new symbol until a preset vocabulary size is reached.
40 more timeline events
All Facts (20)
The BPE algorithm originates from the field of data compression and builds a vocabulary by iteratively merging high-frequency character pairs.
85%VerifiedMainstream tokenizer approaches include BPE (Byte Pair Encoding), WordPiece, and SentencePiece.
80%VerifiedModern LLMs generally use BPE (Byte Pair Encoding) or WordPiece algorithms for tokenization.
80%VerifiedOpenAI uses the BPE (Byte Pair Encoding) algorithm for token encoding.
80%VerifiedBPE（Byte Pair Encoding）是大语言模型使用的Token化算法，通过统计训练语料中字符对的共现频率迭代合并最常见的字符组合
80%VerifiedThe vocabulary of a standard autoregressive language model typically contains 32,000 to 128,000 tokens.
80%VerifiedGPT-series models use the Byte Pair Encoding (BPE) algorithm to segment text into tokens.
80%VerifiedIn GPT-series models, one Chinese character typically corresponds to 1 to 2 tokens, and one English word corresponds to approximately 1 to 1.5 tokens.
80%VerifiedA token is the smallest semantic unit segmented by subword tokenization algorithms such as BPE, and is not simply equivalent to a single character or word.
75%VerifiedModern large models use tokenization algorithms such as BPE or SentencePiece to build vocabularies, with mainstream models having vocabulary sizes ranging from 30,000 to 150,000 tokens.
75%VerifiedThe BPE (Byte Pair Encoding) algorithm commonly used in the GPT series builds a vocabulary by counting the frequency of adjacent character pairs and merging the highest-frequency character pairs.
75%VerifiedGPT series models use byte-level BPE (Byte-level BPE), which takes raw bytes as the basic unit to eliminate the out-of-vocabulary character problem.
75%VerifiedBoth the GPT series and the Claude series employ a tokenization mechanism similar to Byte-Pair Encoding (BPE).
75%VerifiedLarge models process text using Token (word element) as the minimum processing unit, with tokenizers handling the segmentation.
70%VerifiedByte Pair Encoding (BPE) was proposed by Philip Gage in 1994 and was later introduced into the field of neural machine translation by Sennrich et al. in 2016.
70%VerifiedThe BPE (Byte Pair Encoding) algorithm was originally introduced into NLP by Sennrich et al. in 2016, and is essentially a lossless data compression technique.
70%VerifiedBPE (Byte Pair Encoding) was introduced into the NLP field by Sennrich et al. in 2016.
70%VerifiedSubword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
65%VerifiedDifferent models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.
65%VerifiedBPE was originally a data compression algorithm that was introduced into NLP in 2016 for subword segmentation in neural machine translation.
65%