Chonkie
A text chunking library within the Python ecosystem, specifically optimized for tiktoken integration. It delivers strong performance in chunking scenarios that rely on BPE (Byte Pair Encoding) tokenization, making it well-suited for workflows centered on OpenAI-compatible tokenizers.
Timeline (last 90 days)
The knowledge layer of Agentic GraphRAG uses Docling for document ingestion and Chonkie for chunking, preserving document section structure, tables and paragraph hierarchy during chunking, and PDFs also record page number positions.
In the recursive chunking benchmark (1MB, 1000/200), Chunkr achieved 2,264 MB/s, LangChain achieved 769 MB/s, LlamaIndex only 10 MB/s, and Chonkie 225 MB/s.
In the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
Chonkie has been specifically optimized for tiktoken integration, while Chunkr has not yet reached the same level of adaptation at the tokenizer call layer, which is the reason Chunkr lags behind in the BPE scenario.
All Facts (4)
The knowledge layer of Agentic GraphRAG uses Docling for document ingestion and Chonkie for chunking, preserving document section structure, tables and paragraph hierarchy during chunking, and PDFs also record page number positions.
50%UnverifiedIn the recursive chunking benchmark (1MB, 1000/200), Chunkr achieved 2,264 MB/s, LangChain achieved 769 MB/s, LlamaIndex only 10 MB/s, and Chonkie 225 MB/s.
50%UnverifiedIn the BPE tokenization (cl100k_base, 512/50) scenario, Chunkr achieved 38 MB/s, while Chonkie led at 151 MB/s.
50%UnverifiedChonkie has been specifically optimized for tiktoken integration, while Chunkr has not yet reached the same level of adaptation at the tokenizer call layer, which is the reason Chunkr lags behind in the BPE scenario.
50%