vllm — vLLM is a high-throughput, memory-efficient library designed for Large Language Model (LLM) inference and serving.

vLLM is a high-throughput, memory-efficient library designed for Large Language Model (LLM) inference and serving.

About this project

vLLM is a library for LLM inference and serving that focuses on high throughput and efficient memory management. It utilizes PagedAttention to manage attention key and value memory and supports continuous batching, chunked prefill, and prefix caching. Key technical capabilities include: - Performance Optimizations: Support for various quantization methods (FP8, INT8, INT4, AWQ, GGUF, etc.), optimized attention kernels (FlashAttention, FlashInfer), and speculative decoding. - Flexibility: Integration with Hugging Face models, support for streaming outputs, structured output generation, and an OpenAI-compatible API server. - Distributed Inference: Support for tensor, pipeline, data, expert, and context parallelism. - Hardware Compatibility: Works with NVIDIA, AMD, and Intel GPUs, as well as x86/ARM/PowerPC CPUs, and various hardware plugins like Google TPUs and Huawei Ascend. - Model Support: Compatible with over 200 architectures, including decoder-only LLMs, Mixture-of-Experts (MoE), multi-modal models, and embedding/retrieval models.