tiktoken

tiktoken is a fast Byte Pair Encoding (BPE) tokeniser designed for use with OpenAI's models. It converts text into token sequences that language models can process, with reversible and lossless encoding, support for arbitrary text, and compression where each token averages about 4 bytes. The library provides access to specific encodings such as o200k_base and cl100k_base, and can select the appropriate tokeniser for models like gpt-4o via encoding_for_model. It includes an educational submodule for visualising and learning the BPE procedure. Users can extend tiktoken by creating custom Encoding objects directly or by registering new encodings through the tiktoken_ext plugin mechanism. The project reports 3-6x faster performance than a comparable open source tokeniser in its benchmarks.