SentencePiece

Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.

Different models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.

Tokens are generated by the BPE (Byte Pair Encoding) algorithm; in English text, each token corresponds on average to approximately 4 characters (about 0.75 words).

SentencePiece is a language-agnostic tokenization framework proposed by Google that models directly on raw text and is suitable for multilingual use and languages without clear word boundaries.

OpenAI uses the BPE (Byte Pair Encoding) algorithm for tokenization, while Google's SentencePiece adopts a language-agnostic tokenization method trained directly on raw text.

OpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.

llama.cpp's tokenizer implementation follows the two mainstream schemes of SentencePiece and BPE, and provides a unified abstraction for the vocabulary embedded in GGUF, covering details such as special tokens, byte fallback and normalization rules.

When encountering a character that does not exist in the vocabulary, SentencePiece's byte fallback mechanism decomposes the character into a sequence of raw UTF-8 bytes, mapping each byte to a token, thereby avoiding UNK tokens.

TinyLlama's UNK rate is 0.0% on all tested scripts, because SentencePiece's byte fallback mechanism never produces UNK tokens.

The researchers conducted a four-reading audit on another four VLMs covering three tokenizer families, concluding that the Yes/No reading confusion is a protocol-level rather than model-level issue.

Across 9,000 greedy-decoding questions covering all three splits, token ID 3582 (corresponding to the bare word 'yes') and 1217 (corresponding to the bare word 'no') were never once generated by LLaVA-1.5-7B; what the model actually outputs is ' Yes' (token 3869) and ' No' (token 1939).

SentencePiece treats spaces as ordinary characters (denoted by ▁), and when the prompt ends with 'ASSISTANT:' it retains the space byte after the colon, so the next token the model predicts is actually the space-prefixed form ' Yes'/' No'.

Porting LLaVA-1.5's token IDs as-is to two models with disjoint vocabularies drops F1 straight to 0.00, whereas using tokenizer-derived readings keeps F1 stable between 0.80 and 0.92.

ArcLM provides both a word-level tokenizer and SentencePiece support.

The ArcLM core installation automatically pulls in dependencies such as NumPy and SentencePiece, but does not pull PyTorch.

A Tokenizer is responsible for splitting raw text into token sequences that the model can process; mainstream approaches include BPE, WordPiece, and SentencePiece.

Metaspace is a special pre-tokenization processing technique common in the SentencePiece tokenization system, which replaces spaces between words with a special underscore character (▁).

The SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.

The GLM series uses a SentencePiece-based tokenizer and includes several special control tokens such as [gMASK] and <sop>.

Mainstream large language models generally adopt BPE (Byte Pair Encoding) or its variants (such as SentencePiece) as tokenization algorithms, and GPT-4's vocabulary is approximately 100,000 tokens.