multimodalpractice

This is a multimodal model learning project organized in notebook form, following the main thread of "visual foundation models → image-text alignment → multimodal understanding and generation → diffusion models → multimodal large models." Each notebook includes Chinese annotations, making it suitable for both introductory and advanced learning. The content progresses by number: 01 ViT: Hand-write a Vision Transformer from scratch, covering Patch Embedding, learnable positional encoding, CLS Token, Transformer Encoder, and the classification head, with visualizations of patch partitioning and attention heatmaps, while also demonstrating the official HuggingFace API. 02 CLIP: Demonstrate the official API for image-text similarity and zero-shot classification, step by step breaking down text/image encoding, projection, L2 normalization, and temperature-scaled cosine similarity, and hand-write a simplified dual-encoder with contrastive loss. 03 ALBEF: Following the idea of "align first, then fuse," implement a ViT image encoder, BERT text encoder, multi-head cross-attention, and multimodal fusion layer, and explain the three objectives ITC/ITM/MLM along with momentum distillation. 04 BLIP: Introduce the MED unified encoder-decoder architecture, where the same BERT weights switch between three modes—unimodal encoding, cross-modal encoding, and causal decoding—via attention masks, and demonstrate image captioning, visual question answering, feature extraction, and image-text matching. 05 BLIP-2: Use Q-Former to bridge a frozen EVA-ViT-G/14 and a frozen OPT, use hooks to capture internal tensors of Q-Former, show the shapes of 32 learnable Query Tokens, and hand-write a simplified Q-Former. 06 Stable Diffusion architecture visualization: Three sub-notebooks use print(model), torchinfo, and torchviz respectively to display the VAE, CLIP text encoder, Tokenizer, and U-Net, comparing three perspectives: the text tree, the parameter table, and the autograd computation graph. 07 DDIM manual inference: Without relying on the Diffusers Pipeline, implement from scratch timestep embedding, text encoding, cross-attention, the U-Net denoising loop, and VAE decoding, and explain the differences between DDIM and DDPM. 08 Text-to-image practice: Based on diffusers and ModelScope, complete weight download, Pipeline loading, prompt inference, and result visualization. 09 Qwen3-VL: Load Qwen3-VL-2B-Instruct, demonstrate multimodal inference and the complete structure, and explain Patch Merger, DeepStack, and 3D convolutional patch embedding for unified image/video processing. The repository also provides suggested learning paths, a dependency list (torch, transformers, diffusers, modelscope, torchinfo, torchviz, etc.; torchviz requires system-level Graphviz), and quick-start steps; model weights are automatically downloaded on first run and cached in the corresponding subdirectories under model_cache/. The project uses the MIT license.