About this project

labml.ai provides over 60 documented PyTorch implementations of deep learning papers and algorithms, each accompanied by side-by-side explanatory notes rendered on their website. The project aims to help learners understand how these algorithms work through clear, annotated code. The repository covers a wide range of topics: - **Transformers**: Original transformer, Transformer XL, Switch Transformer, Feedback Transformer, ViT, FNet, ALiBi, Rotary Positional Embeddings, MLP-Mixer, gMLP, and more. - **Diffusion Models**: DDPM, DDIM, Latent Diffusion, and Stable Diffusion implementations. - **Generative Adversarial Networks**: Original GAN, DCGAN, CycleGAN, Wasserstein GAN, StyleGAN2. - **Reinforcement Learning**: Proximal Policy Optimization (PPO) with GAE, Deep Q-Networks (DQN) with Dueling Networks, Prioritized Replay, and Double Q. - **Optimizers**: Adam, AMSGrad, AdaBelief, Sophia-G, Rectified Adam, Noam, and others. - **Architectures**: ResNet, Capsule Networks, LSTM, Recurrent Highway Networks, U-Net, Graph Neural Networks (GAT, GATv2), HyperNetworks. - **Normalization**: Batch Norm, Layer Norm, Instance Norm, Group Norm, Weight Standardization, DeepNorm. - **Other topics**: Knowledge distillation, adaptive computation (PonderNet), uncertainty quantification, language model sampling techniques (greedy, temperature, top-k, nucleus), and scalable training with Zero3 memory optimizations. Installation: `pip install labml-nn` The associated website at nn.labml.ai renders the implementations as formatted educational notes.