localai

LocalAI is a self-hosted, open-source AI inference engine designed to run a wide range of models locally without requiring a GPU. It presents itself as a small core rather than a monolithic bundle: each backend wraps an upstream engine (such as llama.cpp, vLLM, whisper.cpp, stable-diffusion, or MLX) in its own container image, pulled only when a model needs it. This means users install only the components their chosen models require. Key characteristics described in the README: - Composable architecture: backends are separate and fetched on demand. - Open and extensible: users can load any model or build a custom backend in any language against an open interface. - Drop-in API compatibility: OpenAI, Anthropic, and ElevenLabs APIs are supported across backends. - Multi-modality: text generation, text-to-audio, audio-to-text, image generation, vision, object detection, reranking, and embeddings behind one API. - Hardware support: NVIDIA (CUDA 12/13), AMD (ROCm), Intel (oneAPI/SYCL), Apple Silicon (Metal), Vulkan, NVIDIA Jetson (L4T), and CPU-only. - Multi-user features: API key authentication, user quotas, and role-based access. - Built-in agents: autonomous agents with tool use, RAG, MCP, and skills, plus an Agent Hub. - Privacy-first positioning: data stays on the user's infrastructure. Installation options include a macOS DMG (unsigned, requiring a quarantine attribute removal) and container images for Docker or podman, with variants for CPU, NVIDIA CUDA 12/13, Jetson ARM64, AMD ROCm, Intel oneAPI, and Vulkan. Models can be loaded from a model gallery, Hugging Face, the Ollama OCI registry, YAML configs, or standard OCI registries. A terminal agent is available for interacting with a running server, supporting commands like /models and /model. The project reports 60+ supported backends and maintains several native C/C++/GGML engines, including vllm.cpp, parakeet.cpp, moss-transcribe.cpp, moss-tts.cpp, magpie-tts.cpp, ced.cpp, voice-detect.cpp, voxtral-tts.c, vibevoice.cpp, rf-detr.cpp, locate-anything.cpp, depth-anything.cpp, face-detect.cpp, free-splatter.cpp, trellis2.cpp, privacy-filter.cpp, LocalVQE, and a local vector store. It also maintains apex-quant, a quantization recipe for Mixture-of-Experts models. Additional features listed include constrained grammars, P2P inferencing, distributed mode with PostgreSQL and NATS, voice activity detection (Silero-VAD), an integrated WebUI, fine-tuning and quantization workflows, WebRTC support, and a Realtime API for speech-to-speech. The project is MIT-licensed, created by Ettore Di Giacinto and maintained by a small team with community contributions.