powerinfer

PowerInfer is an inference runtime for deploying large language models on a personal computer. Its design exploits activation locality: consistently active neurons are kept on the GPU, while input-dependent neurons run on the CPU. Adaptive predictors and neuron-aware sparse operators guide this division, reducing GPU memory pressure and CPU-GPU data transfers. Hybrid inference can use all available VRAM or follow a configurable --vram-budget limit; CPU-only inference is also available. The project uses PowerInfer GGUF files containing model and predictor weights, with tools for converting original model and predictor repositories. It supports INT4 Q4_0 quantization, limited dense-inference mode, local serving, batched generation, and perplexity evaluation. Its command-line examples largely follow llama.cpp usage, although --vram-budget replaces -ngl for PowerInfer offloading. Documented model families include Falcon-40B, ReLU-based Llama 2 variants, ProSparse Llama 2, and Bamboo-7B. Compatibility is not general: the FAQ states that models must use ReLU, ReGLU, or Squared ReLU activations, so unsupported architectures such as Mistral, original Llama, and Qwen are not currently covered. PowerInfer builds with CMake and Python on Linux or Windows with x86-64 AVX2 CPUs, offering NVIDIA CUDA and AMD ROCm/HIP paths as well as CPU-only operation. Apple Silicon CPU inference is listed for macOS, but the README says it is not optimized there and Metal sparse inference remains planned. The README reports benchmark results on specific RTX 4090 and RTX 2080 Ti tests and links to a paper for evaluation details.