swift-tokenizers

Swift Tokenizers is a Swift package that wraps Hugging Face's Rust `tokenizers` crate, providing high-performance tokenization for Swift applications. Unlike Swift Transformers, it focuses only on tokenization and does not depend on the Hugging Face Hub, so model files are downloaded separately and loaded from a local directory. Requirements and platform support: Swift 6.2 or newer (Xcode 26 on Apple platforms), macOS 14+, iOS 17+, and Linux on x86_64 and aarch64. The Rust backend ships as a single SE-0482 staticLibrary artifactbundle containing Apple and Linux slices, so a single dependency works across platforms without conditional manifest logic. Linux binaries target glibc on Ubuntu 22.04 and are forward-compatible with newer Ubuntu LTS images; musl is not supported. Older toolchains cannot resolve the package because the artifact type landed in Swift 6.2. Core capabilities: - Load tokenizers from a local directory containing tokenizer.json and sidecar files such as tokenizer_config.json, config.json and chat_template.jinja. - Encode text to token IDs and decode token IDs back to text. - encodeWithMetadata returns richer encoding data including token strings, masks, sequence indices, word indices and offset spans. - A StreamingDetokenizer emits text chunks as soon as complete Unicode scalars are available, with bounded internal state that does not grow with stream length. It can be seeded with prior tokens when resuming an interrupted stream. - Chat template application via applyChatTemplate, including tool calling with function definitions. - Batch encoding and tokenize operations. The API throws TokenizerError on failure (invalid token IDs, configuration mismatches, internal Rust errors), using typed throws so callers can catch cases exhaustively. A migration section documents the change from silently returning empty results to throwing, and another section covers migrating from Swift Transformers, including replacing the package dependency and switching from hub-based loading to local-directory loading. The README includes a benchmark table comparing Swift Tokenizers 0.5.0 against Swift Transformers 1.3.2 on an M3 MacBook Pro, reporting faster tokenizer load, tokenization, decoding, and model loading times. Benchmarks are gated behind the TOKENIZERS_ENABLE_BENCHMARKS environment variable so ordinary consumers do not pull in mlx-swift-lm, which requires Metal and is macOS-only. Model loading benchmarks require Metal and must run through xcodebuild; tokenizer-only benchmarks can run with swift test in release configuration.