trl — TRL is a comprehensive library for post-training foundation models using reinforcement learning and preference optimization techniques, built on the Hugging Face ecosystem.

TRL is a comprehensive library for post-training foundation models using reinforcement learning and preference optimization techniques, built on the Hugging Face ecosystem.

About this project

TRL (Transformers Reinforcement Learning) is a library designed for the post-training of foundation models. It provides a suite of trainers to implement various fine-tuning and alignment methods, including: - Supervised Fine-Tuning (SFT) via `SFTTrainer` - Group Relative Policy Optimization (GRPO) via `GRPOTrainer` - Direct Preference Optimization (DPO) via `DPOTrainer` - Kahneman-Tversky Optimization (KTO) via `KTOTrainer` - Reward model training via `RewardTrainer` The library is integrated with the Hugging Face ecosystem, leveraging Accelerate for scaling across multi-node clusters (supporting DDP, DeepSpeed, and FSDP) and PEFT for efficient training using LoRA/QLoRA and quantization. It also integrates with Unsloth for kernel-level acceleration. Users can interact with TRL through a Python API or a Command Line Interface (CLI) for code-free fine-tuning. The library also includes guides for training on long context sequences (beyond 1M tokens).