megatron-lm

Megatron-LM is a reference example that bundles Megatron Core with pre-configured training scripts, aimed at research teams, learning distributed training, and quick experimentation. Megatron Core is a composable library of GPU-optimized building blocks for custom training frameworks, offering transformer components, advanced parallelism strategies (tensor, pipeline, data, expert, and context parallelism), mixed precision support (FP16, BF16, FP8, and FP4), and model architectures. The repository also includes inference engines, model export utilities, post-training tools such as quantization and distillation, and reinforcement learning support including RLHF. Megatron Bridge provides bidirectional checkpoint conversion between Hugging Face and Megatron formats. The project reports training models from 2B to 462B parameters across thousands of GPUs, with up to 47% Model FLOP Utilization on H100 clusters. Installation is available through PyPI or from source, and documentation covers first training runs, parallelism strategies, and contribution guidelines.