bitnet

bitnet.cpp is the official inference framework for 1-bit LLMs such as BitNet b1.58. It provides a suite of optimized kernels intended to enable fast and lossless inference of 1.58-bit models on CPU and GPU, with NPU support noted as forthcoming. The project is based on the llama.cpp framework, and its kernels build on lookup-table methods pioneered in T-MAC. Key capabilities described in the README: - CPU inference for ternary/1-bit models, with reported speedups of 1.37x to 5.07x on ARM CPUs and 2.37x to 6.17x on x86 CPUs versus full-precision models, plus reported energy reductions of 55.4%-70.0% (ARM) and 71.9%-82.2% (x86). - Ability to run a 100B BitNet b1.58 model on a single CPU at speeds described as comparable to human reading (5-7 tokens per second). - Official GPU inference kernel released in May 2025. - Quantization types including I2_S and TL1, with kernel support varying by model and architecture (x86 vs ARM). - Model conversion from .safetensors checkpoints to GGUF via a helper script. - Benchmarking utilities (e2e_benchmark.py) and a dummy-model generator for unsupported layouts. - Chat/conversation mode for instruct models. Official models highlighted include BitNet-b1.58-2B-4T (2.4B parameters, trained on 4T tokens), BitNet-embedding-0.6B, and BitNet-embedding-270M. Community models listed as supported include bitnet_b1_58-large, bitnet_b1_58-3B, Llama3-8B-1.58-100B-tokens, and the Falcon3 and Falcon-E families. Installation requires Python 3.10+, CMake 3.22+, and Clang 18+, with conda recommended. Build steps involve cloning the repository recursively, installing Python dependencies, downloading a model, and running setup_env.py with a chosen quantization type. Inference is run through run_inference.py with options for prompt, token count, threads, context size, temperature, and conversation mode. The README also includes an FAQ covering common build issues, such as std::chrono errors in llama.cpp and configuring clang within a conda environment on Windows. The project is released under the MIT license.