tiny-llm

tiny-llm is a hands-on course for systems engineers who want to understand LLM inference end to end. It is positioned as an LLM-serving counterpart to CMU's Needle project: learners build the path that loads a Qwen3 model, turns tokens into logits, and generates text. The course starts with array and matrix operations, then introduces kernels and serving machinery as the running model needs them. The implementation is kept small enough to read end to end, connecting equations to memory traffic, kernel occupancy, KV-cache growth, batching, and request scheduling. It is built on MLX arrays and the MLX extension runtime without high-level neural-network layers. When a chapter teaches an operator, learners implement that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX serves as the correctness oracle and performance baseline. The four-week learning path covers: - Week 1: building a Qwen3 model from mlx.core array operations, including attention, RoPE, GQA, RMSNorm, MLP, sampling, and the autoregressive loop. - Week 2: making a single request faster with kv-cache, capacity-cache, packed W4 weights, SIMD matrix prefill, fused primitives, and tiled prefill. - Week 3: building a mini vLLM with continuous batching, chunked admission, paged KV cache, direct paged attention, and paged FlashAttention. - Week 4: building a coding agent with a validated agent loop, workspace inspection, approved edits, validation commands, checkpoint-and-resume, context compaction, inspect-and-steer pauses, outcome evaluation, forked steered branches, and bounded tool evidence. The project targets Apple Silicon because it provides a practical local environment with one shared memory space and direct access to Metal kernels. Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache costs while remaining small enough for local iteration. The book is published at skyzh.github.io/tiny-llm. The repository contains a tiny_llm package for student implementations and tiny_llm_ref with the reference solution. A roadmap table tracks implementation, tests, documentation, and editorial audit status for each chapter.