tokenizer

Different models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.

GPT-3 has a vocabulary of approximately 50,000 tokens.

中文字符在不同tokenizer下的Token数量差异可达30%-50%

When models iterate, developers need to watch details such as tokenizer changes and context window adjustments, and re-validate API interfaces, prompt adaptation and evaluation pipelines.

When encoding text, "cat", " cat", "Cat", and "cats" may correspond to completely different token IDs, and a single word may be split into multiple tokens.

Different models may use different tokenizers, and a token ID from one model may not apply to another model.

Once a tokenizer is trained it is fixed, and model training and inference must use the same tokenizer; otherwise the correspondence between tokens and model parameters would be broken.

Different model families (such as the GPT series and the Llama series) have different tokenizers, and the same sentence may differ noticeably in token count and segmentation across models.

The smallest unit most LLMs process is the token, which can be a complete word, part of a word, a punctuation mark or another text fragment, with the specific segmentation depending on the tokenizer.

If a language uses rare Unicode symbols, unconventional whitespace structures, or deliberately constructed ambiguous tokens, the tokenizer may split them into inefficient or even meaningless fragments, significantly increasing the difficulty of model understanding and generation.

Perplexity depends heavily on the tokenizer: different tokenizers split text into different token granularities, so the same passage yields different token counts, and perplexity values cannot be compared directly.

The reliability of offsets depends on how the tokenizer maps to character indices; if token indices are returned rather than character indices, an extra alignment step is needed, especially in multi-byte character scenarios.

There is no universal conversion ratio between the character count of a repository and the model's token count; the ratio is model-dependent and drifts with language and coding style.

Using another vendor's tokenizer library to compute the target model's token count leads to systematic bias, because different models use different vocabularies, and this bias is unobservable from the caller's perspective.

Changes to the tokenizer's vocabulary size, special token definitions, or context handling methods will cause some preprocessing and postprocessing logic to need updating.

human-tps uses real tokenizers rather than counting by character or word to measure token speed.

Token segmentation logic varies by model; the handling of Chinese, code, and punctuation each differs, and a passage of English text may be split into far more tokens than the number of words

Using real tokenizers for measurement places human t/s and model t/s in a comparable context.

In Apple's dictation technical pipeline, speech first passes through a tokenizer acting as an encoder that maps short-time waveform windows into representations readable by the language model, which then processes these representations.

In on-device scenarios, the encoder's parameter count directly affects power consumption and latency—more parameters means greater DRAM usage, which impacts both energy consumption and response speed.