cl100k_base
A token encoding scheme in the tiktoken library used for the GPT-3.5/4 series, mapping text to token sequences. It differs from the tokenizer used by Claude, so counting Claude tokens with it can only be used for relative ranking.
Core Facts
The GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).
The cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.
OpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
Timeline (last 90 days)
tiktoken is OpenAI's open-source tokenizer library, and cl100k_base is its commonly used encoding scheme; an English word typically corresponds to 1-2 tokens, while Chinese uses more.
The authors reproduced the previously published "The Tokenizer Tax" table, with all 66 of 66 cells within 0.5% error; the average tax rate is 8.03x under cl100k_base (8.0x in the original paper) and 2.14x under o200k_base (2.1x in the original paper).
o200k_base is the tokenization encoding used by newer GPT series, with a larger vocabulary, and its segmentation efficiency for structured text such as code and JSON is usually better than the older cl100k_base.
o200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is clearly better than the earlier cl100k_base.
In a test with a nested configuration object, encoded using OpenAI's cl100k_base tokenizer, JSON was 99 tokens and 322 characters, while YAML was 66 tokens and 223 characters, with YAML reducing the token count by 33.3%.
In a test with regular tabular data of 10 rows and 4 fields per row, encoded using the cl100k_base tokenizer, JSON was 323 tokens and 963 characters, while YAML was 221 tokens and 590 characters, with YAML ahead by 31.6%; the TOON format was 97 tokens and 235 characters, 70% fewer than JSON and 56.1% fewer than YAML.
tiktoken is OpenAI's open-source tokenizer library, cl100k_base is the encoding scheme used for the GPT-3.5/4 series, and its tokenization results differ from Anthropic Claude's proprietary tokenizer, usually by a single-digit to low-teen percentage.
The o200k_base vocabulary is expanded to approximately 200,000 tokens, offering more efficient encoding of common symbols and multi-byte characters in code compared to cl100k_base.
cl100k_base is the specific vocabulary used by models such as GPT-4 and text-embedding-ada-002.
In the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
3 more timeline events
All Facts (15)
The GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).
90%VerifiedThe cl100k_base vocabulary used by GPT-4 contains approximately 100,000 token units, and the number of tokens in a text can be precisely calculated using the tiktoken library.
80%VerifiedOpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
65%Unverifiedtiktoken is OpenAI's open-source tokenizer library, and cl100k_base is its commonly used encoding scheme; an English word typically corresponds to 1-2 tokens, while Chinese uses more.
50%UnverifiedThe authors reproduced the previously published "The Tokenizer Tax" table, with all 66 of 66 cells within 0.5% error; the average tax rate is 8.03x under cl100k_base (8.0x in the original paper) and 2.14x under o200k_base (2.1x in the original paper).
50%Unverifiedo200k_base is the tokenization encoding used by newer GPT series, with a larger vocabulary, and its segmentation efficiency for structured text such as code and JSON is usually better than the older cl100k_base.
50%Unverifiedo200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is clearly better than the earlier cl100k_base.
50%UnverifiedIn a test with a nested configuration object, encoded using OpenAI's cl100k_base tokenizer, JSON was 99 tokens and 322 characters, while YAML was 66 tokens and 223 characters, with YAML reducing the token count by 33.3%.
50%UnverifiedIn a test with regular tabular data of 10 rows and 4 fields per row, encoded using the cl100k_base tokenizer, JSON was 323 tokens and 963 characters, while YAML was 221 tokens and 590 characters, with YAML ahead by 31.6%; the TOON format was 97 tokens and 235 characters, 70% fewer than JSON and 56.1% fewer than YAML.
50%Unverifiedtiktoken is OpenAI's open-source tokenizer library, cl100k_base is the encoding scheme used for the GPT-3.5/4 series, and its tokenization results differ from Anthropic Claude's proprietary tokenizer, usually by a single-digit to low-teen percentage.
50%UnverifiedThe o200k_base vocabulary is expanded to approximately 200,000 tokens, offering more efficient encoding of common symbols and multi-byte characters in code compared to cl100k_base.
50%Unverifiedcl100k_base is the specific vocabulary used by models such as GPT-4 and text-embedding-ada-002.
50%UnverifiedIn the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
50%UnverifiedThe GPT series uses the BPE algorithm with vocabularies such as cl100k_base, the Llama series uses SentencePiece, and Gemini has its own proprietary tokenization scheme.
50%UnverifiedGPT-4 使用 cl100k_base tokenizer，与 Claude 的 tokenizer 对中文、代码等内容的切分策略不同
50%