llama.cpp — A C/C++ implementation for LLM and VLM inference designed for minimal setup and high performance across diverse hardware.

A C/C++ implementation for LLM and VLM inference designed for minimal setup and high performance across diverse hardware.

About this project

llama.cpp provides a plain C/C++ implementation for Large Language Model (LLM) and Vision Language Model (VLM) inference without external dependencies. It is built on the ggml library and supports a wide array of hardware architectures, including Apple Silicon (via Metal, ARM NEON, and Accelerate), x86 (AVX, AVX2, AVX512, AMX), and RISC-V. Key capabilities include: - Quantization: Supports 1.5-bit through 8-bit integer quantization to reduce memory usage and increase inference speed. - Hardware Acceleration: Custom CUDA kernels for NVIDIA GPUs, as well as support for AMD (HIP), Moore Threads (MUSA), Vulkan, SYCL, and OpenCL. - Hybrid Inference: Ability to use CPU+GPU hybrid inference for models that exceed available VRAM. - Tools: Includes a command-line interface (cli), a completion tool, and an OpenAI-compatible REST API server with a built-in web UI. - Broad Backend Support: Compatible with BLAS, BLIS, CANN, Metal, WebGPU, and various other hardware-specific backends.