planckgpt

PlanckGPT is an educational project that demonstrates how to train a GPT-style language model from scratch on consumer-level hardware. The model has approximately 540 million parameters and is pretrained on roughly 2 billion tokens from the Fineweb-edu dataset, with an optional chat finetuning stage using Smol-smoltalk data. The project targets achieving the best quality-to-training-time ratio, with a soft limit of 50 hours of training time. On the author's laptop (Intel Core i7 14700HX, 24GB RAM, RTX 5070 Mobile 8GB), pretraining takes about 28 hours and 20 minutes, while chat finetuning takes roughly 4 hours and 10 minutes. The architecture uses a 14-layer transformer decoder with 7 query heads, 896-dimensional embeddings, rotary positional embeddings, alternating value embeddings, squared ReLU activation, and RMSNorm. The tokenizer is Tiktoken with GPT-2 encoding (50,257 vocabulary size). Training employs NorMuon optimizer for transformer weights, 8-bit Adam for embeddings, BF16 mixed precision, gradient checkpointing, and torch.compile. The project provides scripts for pretraining, evaluation, chat finetuning, and inference. Pretrained performance is close to GPT-2-medium with ~2.97 average validation loss on Fineweb-edu. Chat-finetuned performance reaches ~1.13 average validation loss on Smol-smoltalk. Inference uses top-k and top-p sampling with KV cache for faster generation. The project is inspired by modded-nanogpt and nanochat, and is licensed under Apache 2.0.