nanogpt

# nanoGPT A minimal, readable, and fast repository for training and fine-tuning medium-sized GPT (Generative Pre-trained Transformer) models. This is a rewrite of minGPT, redesigned to prioritize practical performance while keeping the code minimal and readable. ## What's Inside The core `train.py` script (~300 lines) implements a complete training loop using PyTorch DDP. It reproduces GPT-2 (124M) on OpenWebText on a single 8x A100 40GB node in approximately 4 days. The repository also includes `sample.py` for text generation from trained or pretrained models. ## Key Capabilities - **Training**: Complete training loop with distributed data parallel (DDP) support via `torchrun`. Reproduce GPT-2 (124M) on OpenWebText on 8x A100 40GB in ~4 days. - **Fine-tuning**: Initialize from OpenAI GPT-2 checkpoints (gpt2, gpt2-medium, gpt2-large, gpt2-xl) and fine-tune on custom datasets such as Shakespeare text. - **Sampling/inference**: The `sample.py` script generates text with configurable prompts, number of samples, and token limits. - **Efficiency**: Uses PyTorch 2.0's `torch.compile()` by default for significant speedups (e.g., cutting iteration time from ~250ms to ~135ms on A100). - **Multi-GPU/multi-node**: Distributed training via `torchrun` with support for multi-node setups. - **Flexible configuration**: Hyperparameters can be tuned via config files in the `config/` directory or command-line arguments. - **CPU/MPS support**: Works on CPU for experimentation, and Metal Performance Shaders (MPS) on Apple Silicon for 2-3x acceleration. ## Example Workflows Train a character-level GPT on Shakespeare data in minutes on a single GPU, or reproduce GPT-2 benchmarks on OpenWebText with larger hardware. ## Dependencies PyTorch, NumPy, Hugging Face transformers, datasets, tiktoken, wandb, and tqdm. ## Status As of November 2025, nanoGPT has been superseded by nanochat and is considered deprecated, though retained for reference. ## Project Context This project is a community-maintained educational repository for understanding and reproducing GPT-style language model training at scale. It demonstrates that high-quality language model training can be implemented in a few hundred lines of clean, readable code.