exo

exo is an open-source project maintained by exo labs that turns a set of devices into a local AI cluster. Its stated goal is running frontier AI models locally, including models larger than would fit on a single device. The README emphasizes automatic device discovery, so devices running exo find each other without manual configuration. Key documented capabilities: - Automatic device discovery across machines running exo. - RDMA over Thunderbolt 5 support, described as day-0, with the README claiming a 99% reduction in latency between devices. - Topology-aware auto parallel: the system chooses how to split a model across available devices based on device resources and network latency/bandwidth. - Tensor parallelism, with the README stating up to 1.8x speedup on 2 devices and 3.2x on 4 devices. - MLX as the inference backend, with MLX distributed for communication. - API compatibility with OpenAI Chat Completions, Claude Messages, OpenAI Responses, and Ollama APIs. - Custom model loading from the HuggingFace hub. - A built-in dashboard for cluster management and chatting with models, served at http://localhost:52415. Installation paths described in the README include running from source on macOS and Linux, a Nix flake command, and a macOS app distributed as a DMG or via Homebrew cask. On macOS the project uses the GPU; on Linux it currently runs on CPU, with GPU support noted as under development. The README also documents environment variables for model directories, offline mode, image models, cluster namespace isolation, and tracing, plus a benchmarking tool called exo-bench. Benchmark images in the README are attributed to Jeff Geerling and show Qwen3-235B, DeepSeek v3.1 671B, and Kimi K2 Thinking running on four M3 Ultra Mac Studio machines with tensor parallel RDMA. These are third-party measurements rather than independent verification by this directory. Licensing is Apache-2.0. The project provides a Discord community and an X account for updates.