nanogpt

nanoGPT is a lightweight, high-performance repository for training and fine-tuning medium-sized GPT (Generative Pre-trained Transformer) models. It is a rewrite of minGPT, redesigned to prioritize practical performance while keeping the code minimal and readable. Key capabilities: - **Training**: The core `train.py` script (~300 lines) implements a complete training loop. It reproduces GPT-2 (124M) on OpenWebText on a single 8x A100 40GB node in approximately 4 days using PyTorch DDP. - **Fine-tuning**: Supports initializing from pretrained OpenAI GPT-2 checkpoints (gpt2, gpt2-medium, gpt2-large, gpt2-xl) and fine-tuning on custom datasets such as Shakespeare text. - **Sampling/inference**: The `sample.py` script generates text from trained or pretrained models with configurable prompts, number of samples, and token limits. - **Efficiency**: Uses PyTorch 2.0's `torch.compile()` by default for significant speedups (e.g., cutting iteration time from ~250ms to ~135ms on A100). - **Multi-GPU/multi-node**: Distributed training via `torchrun` with support for multi-node setups. - **Flexible configuration**: Hyperparameters can be tuned via config files in the `config/` directory or command-line arguments. - **CPU/MPS support**: Works on CPU for experimentation, and Metal Performance Shaders (MPS) on Apple Silicon for 2-3x acceleration. Example workflows include training a character-level GPT on Shakespeare data in minutes on a single GPU, or reproducing GPT-2 benchmarks on OpenWebText with larger hardware. Dependencies: PyTorch, NumPy, Hugging Face transformers, datasets, tiktoken, wandb, and tqdm. Note: As of November 2025, nanoGPT has been superseded by nanochat and is considered deprecated, though retained for reference.