model-optimizer

NVIDIA Model Optimizer (ModelOpt) is a unified open-source library for compressing and accelerating deep learning models. It supports Hugging Face, PyTorch, and ONNX model inputs and exports optimized quantized checkpoints ready for deployment in frameworks like TensorRT-LLM, TensorRT, vLLM, and SGLang. Key techniques include: - Post-Training Quantization (PTQ): Compresses models 2x–4x with formats including FP8, INT8, and NVFP4, supporting LLMs, VLMs, Diffusers, and ONNX models. - Quantization-Aware Training (QAT) / Distillation: Refines accuracy of quantized models through additional training steps. - Pruning: Reduces model parameters and memory footprint by removing unnecessary weights, including heterogeneous pruning via the Puzzletron algorithm. - Distillation: Teaches smaller models to mimic larger ones for reduced deployment size. - Speculative Decoding: Trains draft modules to predict extra tokens during inference, reducing latency. - Sparsity: Compresses models by storing only non-zero parameter values and their locations. - Neural Architecture Search (NAS): Discovers efficient model architectures. The library integrates with NVIDIA Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate for training-required optimization workflows. It also provides AI agent skills for Claude Code and Codex. Installation is via pip (`pip install -U nvidia-modelopt[all]`) or from source. Pre-quantized checkpoints for models like Llama, DeepSeek-R1, and Nemotron are available on Hugging Face. The project follows a structured deprecation policy with a ~1-release migration period.