mlx-serve

mlx-serve is a native server for running large language models locally on Apple Silicon Macs. It is written in Zig and requires no Python at runtime, shipping as a small binary alongside an optional signed macOS menu-bar app called MLX Core. Model support covers native MLX architectures (Gemma, Qwen, Llama, Mistral, DeepSeek, Hunyuan and others) as well as the broader GGUF ecosystem through an embedded llama.cpp, routed automatically by format. Models can be pulled from Hugging Face by short name, org/repo id, or name:tag, and are loaded on demand by name. On the API side, the server exposes several compatible surfaces on one port: OpenAI chat/completions and Responses (including a WebSocket transport), Anthropic Messages, and the Ollama API. This means existing clients and coding agents that speak any of these protocols can point at the local endpoint without modification. Streaming, tool calling with schema-driven repair, JSON-schema constrained decoding, logprobs, vision input and reasoning content are among the supported serving features. Beyond text, the same server offers endpoints for image, video, audio/speech (with voice cloning), music and 3D model generation, with several model options listed per modality and approximate memory requirements. A built-in web console provides a chat playground, monitoring and media tools. The MLX Core app adds multi-session chat, an agent mode with built-in tools and MCP support, an isolated Linux VM sandbox for agent shell commands, a model browser with resumable downloads, voice mode, scheduled tasks, LAN model sharing between Macs, and a settings window for server flags. The README also documents speculative decoding variants, KV-cache quantization, continuous batching and caching, and links to benchmark and performance documentation. Installation is available through Homebrew casks or by building from source with Xcode and the Metal toolchain. The project is MIT licensed and bundles third-party components under their own licenses, with attributions listed in a NOTICE file.