rl

TorchRL is a modular, primitive-first, Python-first PyTorch library for reinforcement learning. It provides composable building blocks for RL systems rather than a single algorithm implementation, keeping code close to the PyTorch programming model. The library is organized around TensorDict, a dictionary-like tensor container that preserves named fields, structure, batch dimensions, and devices throughout the training loop. Key components include environments and transforms (with wrappers for Gymnasium, DM Control, Brax, PettingZoo, VMAS, Isaac Lab, and others), collectors for single-process, async, multiprocess, and distributed execution, replay buffers with prioritized and memmap-backed storage, policy modules with explicit input/output keys, and loss modules covering PPO, SAC, DQN, TD3, REDQ, IQL, CQL, Decision Transformer, Dreamer, MAPPO, IPPO, QMIX/VDN, and more. The library also supports multi-agent, model-based, imitation learning, and LLM post-training workflows including GRPO and SFT objectives. TorchRL 0.13 adds faster recurrent RL paths with Triton and scan backends, custom MuJoCo environments, multi-agent value normalization utilities, HER replay buffers, and improved collector and replay-buffer ergonomics. Installation is available via PyPI with optional extras for Gymnasium, Atari, offline data, multi-agent environments, and LLM backends.