llm-course

The LLM Course is an open educational resource by Maxime Labonne covering the full spectrum of Large Language Model knowledge, from foundational theory to practical engineering. ## Structure The course is divided into three main parts: 1. **LLM Fundamentals** — Optional background covering mathematics for machine learning (linear algebra, calculus, probability), Python data science libraries, neural network fundamentals, and natural language processing basics including tokenization, embeddings, and RNNs. 2. **The LLM Scientist** — Focuses on building LLMs using modern techniques, covering Transformer architecture, tokenization, attention mechanisms, sampling strategies, distributed pre-training, post-training datasets, supervised fine-tuning, preference alignment (DPO/ORPO), and evaluation. 3. **The LLM Engineer** — Focuses on creating and deploying LLM-based applications, covering local model deployment with Ollama and text-generation-webui, RAG pipelines, AI agents, and production monitoring. ## Colab Notebooks The repository includes numerous interactive Google Colab notebooks organized by topic: **Tools**: LLM AutoEval, LazyMergekit (model merging), LazyAxolotl (cloud fine-tuning), AutoQuant (quantization in GGUF/GPTQ/EXL2/AWQ/HQQ), Model Family Tree visualization, ZeroSpace (free Gradio deployment), AutoAbliteration, and AutoDedup. **Fine-tuning**: Notes for fine-tuning Llama 3.1 with Unsloth, Llama 3 with ORPO, Mistral-7b with DPO and QLoRA, and CodeLlama with Axolotl. **Quantization**: Guides for 8-bit quantization, GPTQ 4-bit quantization, GGUF with llama.cpp, and ExLlamaV2. **Other**: Merging models with MergeKit, creating MoEs, abliteration, knowledge graph augmentation for ChatGPT, and decoding strategies. ## Related Resources - A companion book, *LLM Engineer's Handbook*, expands on the course material with end-to-end application coverage. - A DeepWiki version provides a more comprehensive narrative format. - The author maintains additional repos including [llm-datasets](https://github.com/mlabonne/llm-datasets) for post-training data and [awesome-llm-training](https://github.com/mlabonne/awesome-llm-training) for training resources.