llms-from-scratch

This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka. It provides step-by-step PyTorch implementations for developing, pretraining, and finetuning a GPT-like LLM, without relying on external LLM libraries. The main chapters cover text data processing and BPE tokenization, attention mechanisms, implementing a GPT model, pretraining on unlabeled data, finetuning for text classification, and instruction finetuning. Each chapter includes Jupyter notebooks, summarized Python scripts, and exercise solutions. Appendices cover PyTorch basics, LoRA parameter-efficient finetuning, and training-loop enhancements. The repository also contains bonus material such as tokenizer comparisons, KV cache, grouped-query attention, mixture-of-experts, and from-scratch implementations of Llama 3.2, Qwen3, Gemma 3, and other architectures. Code is designed to run on conventional laptops and can use GPUs if available. The author also mentions a companion video course and a sequel book on building reasoning models, as well as links to discussion forums and citation details.