sglang — SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models, designed for low-latency and high-throughput inference.

SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models, designed for low-latency and high-throughput inference.

About this project

SGLang is a serving infrastructure for large language and multimodal models that supports deployments ranging from single GPUs to large distributed clusters. Key technical capabilities include: - Fast Runtime: Features RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, and various parallelism strategies (tensor, pipeline, expert, and data). - Quantization & Optimization: Supports FP4, FP8, INT4, AWQ, and GPTQ quantization, as well as structured outputs and multi-LoRA batching. - Broad Model Support: Compatible with Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, and Mistral, as well as embedding, reward, and diffusion models (e.g., WAN, Qwen-Image). It is compatible with OpenAI APIs and most Hugging Face models. - Hardware Compatibility: Runs on NVIDIA GPUs, AMD GPUs, Intel Xeon CPUs, Google TPUs, and Ascend NPUs. - RL Integration: Serves as a rollout backend for post-training frameworks including AReaL, Miles, slime, Tunix, and verl.