Product

js-tiktoken

A JavaScript implementation of TikToken used for token counting and text encoding for OpenAI models, embedding relatively large BPE ranking-table data.

Timeline (last 90 days)

Oct 20

js-tiktoken embeds a 5.3MB base64 BPE rank table, and the first call inflates the V8 heap by +43MB.

Unverified50%

Oct 13

OpenAI's tokenizer is open source; in JavaScript, js-tiktoken can run in any environment (including the browser), and in Python the corresponding package is tiktoken.

Unverified50%

All Facts (2)

Unverified

js-tiktoken embeds a 5.3MB base64 BPE rank table, and the first call inflates the V8 heap by +43MB.

50%Unverified

OpenAI's tokenizer is open source; in JavaScript, js-tiktoken can run in any environment (including the browser), and in Python the corresponding package is tiktoken.

50%

Source Articles

Migrating 100K Lines of Deno Code to Cloudflare Workers in PracticeOct 20