accelerate

🤗 Accelerate is a library from Hugging Face designed for PyTorch users who want to run their training scripts on any device or distributed configuration without rewriting boilerplate code. It abstracts only the necessary parts—multi-GPU, TPU, and mixed precision (fp8, fp16, bf16)—while leaving the rest of your code unchanged. With just a few lines added to a standard PyTorch script, you can run on single CPU, single GPU, multi-GPU, or TPU setups, with or without mixed precision. The library provides an `Accelerator` class that handles device placement, mixed precision, and distributed training setup. It also includes a CLI tool (`accelerate config` and `accelerate launch`) to quickly configure and launch training environments, eliminating the need to remember `torch.distributed.run` or TPU-specific launchers. For notebook users, a `notebook_launcher` function enables distributed training from Colab or Kaggle notebooks. Accelerate supports DeepSpeed and PyTorch Fully Sharded Data Parallel (FSDP) for advanced distributed training, as well as Megatron-LM. It is used as the backend for Hugging Face Transformers' PyTorch training and is integrated into many other frameworks like fastai, Catalyst, and Stable Diffusion Web UI. Installation is straightforward via `pip install accelerate`, and it requires Python 3.8+ and PyTorch 1.10.0+. The project is open-source under the Apache 2.0 license and has extensive documentation and examples.