tiktoken
A tokenization library developed by OpenAI, based on the Byte Pair Encoding (BPE) algorithm. It is used for text tokenization in models such as ChatGPT.
Core Facts
Token是大语言模型处理文本的基本单位，大约对应0.75个英文单词或1.5个汉字
The GPT series uses tiktoken (based on BPE encoding) for tokenization.
OpenAI provides the open-source tool tiktoken, which allows developers to estimate token counts before calling the API.
Timeline (last 90 days)
The project provides a free counter at modelmarket.dev/tools/tool-list-cost that uses the same tokenizer as the statistics and was compared against tiktoken on 120 of HISTOR's tool lists (2.6 million tokens in total), with zero differences.
tiktoken is OpenAI's open-source tokenizer library, and cl100k_base is its commonly used encoding scheme; an English word typically corresponds to 1-2 tokens, while Chinese uses more.
The OpenAI API's logit_bias parameter does not accept the word itself as a key but requires token ID strings, which must be obtained by encoding with the tokenizer matching the model (such as the tiktoken library).
An LLM gateway needs to parse the model field in the request payload to make routing decisions, use tokenizers such as tiktoken to count tokens in messages for token-based rate limiting, translate OpenAI-format tools definitions into Anthropic's input_schema, and extract and accumulate the usage field from each chunk of the response stream to compute cost.
The author's measurements show that the JSON Schema definitions of 18 tools consume 2,388 tokens, averaging 133 tokens per tool, and in a 10-step Agent run they can account for 36% of input tokens and 32% of the total bill.
When counting with the tiktoken library and the o200k_base encoding, the smallest git_status tool (no parameters) is only 54 tokens, while the largest search_code tool (five parameters) reaches 193 tokens.
tiktoken's README gives an average of about 4 bytes of text per token.
OpenAI's tokenizer is open source; in JavaScript, js-tiktoken can run in any environment (including the browser), and in Python the corresponding package is tiktoken.
tiktoken's encode raises an error when processing text containing special tokens, unless disallowed_special=() is passed.
tiktoken is a compiled package and downloads vocabulary files, so it is not always suitable for lightweight libraries, network-free CI environments, or restricted environments.
16 more timeline events
All Facts (20)
Token是大语言模型处理文本的基本单位，大约对应0.75个英文单词或1.5个汉字
90%VerifiedThe GPT series uses tiktoken (based on BPE encoding) for tokenization.
80%VerifiedOpenAI provides the open-source tool tiktoken, which allows developers to estimate token counts before calling the API.
80%VerifiedOpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
65%VerifiedOpenAI uses the tiktoken tokenizer for token metering.
65%VerifiedThe number of tokens consumed by the same Chinese text may differ by 30%–50% across different large model service providers.
65%UnverifiedLLaMA 3的分词器从SentencePiece升级为基于tiktoken的BPE分词器，词表大小扩展至128,256
90%UnverifiedOpenAI's GPT series uses the tiktoken library, with vocabulary sizes typically ranging between 50,000 and 100,000 Tokens
85%UnverifiedOpenAI uses the tiktoken tokenization algorithm, and Google uses the SentencePiece tokenization algorithm.
60%Unverifiedtiktoken is OpenAI's open-source tokenizer library, and cl100k_base is its commonly used encoding scheme; an English word typically corresponds to 1-2 tokens, while Chinese uses more.
50%UnverifiedThe OpenAI API's logit_bias parameter does not accept the word itself as a key but requires token ID strings, which must be obtained by encoding with the tokenizer matching the model (such as the tiktoken library).
50%UnverifiedAn LLM gateway needs to parse the model field in the request payload to make routing decisions, use tokenizers such as tiktoken to count tokens in messages for token-based rate limiting, translate OpenAI-format tools definitions into Anthropic's input_schema, and extract and accumulate the usage field from each chunk of the response stream to compute cost.
50%UnverifiedWhen counting with the tiktoken library and the o200k_base encoding, the smallest git_status tool (no parameters) is only 54 tokens, while the largest search_code tool (five parameters) reaches 193 tokens.
50%UnverifiedThe author's measurements show that the JSON Schema definitions of 18 tools consume 2,388 tokens, averaging 133 tokens per tool, and in a 10-step Agent run they can account for 36% of input tokens and 32% of the total bill.
50%Unverifiedtiktoken's README gives an average of about 4 bytes of text per token.
50%UnverifiedOpenAI's tokenizer is open source; in JavaScript, js-tiktoken can run in any environment (including the browser), and in Python the corresponding package is tiktoken.
50%UnverifiedPlacing the tiktoken import inside the constructor rather than at the top of the module enables graceful degradation: importing the module itself requires no dependency, and ImportError only appears when a feature that cannot be satisfied is requested.
50%Unverifiedtiktoken is a compiled package and downloads vocabulary files, so it is not always suitable for lightweight libraries, network-free CI environments, or restricted environments.
50%UnverifiedNever use heuristics to bill customers or enforce hard context limits; scenarios with consequences must use tiktoken.
50%UnverifiedThe factory function's auto mode prefers exact tiktoken counting and silently degrades to a heuristic method.
50%