cogvideo

CogVideoX is an open-source video generation model originating from QingYing. The repository provides multiple models including CogVideoX-2B, CogVideoX-5B, CogVideoX-5B-I2V, and the latest CogVideoX1.5-5B series. These models support three core tasks: text-to-video generation, image-to-video generation, and video continuation. The CogVideoX1.5-5B models support 10-second videos at resolutions up to 1360x768, while earlier models generate 6-second videos at 720x480 resolution. The I2V (image-to-video) variant supports variable resolutions with minimum dimension of 768 pixels. Models accept English prompts with token limits of 224-226 tokens. Inference is supported through two frameworks: Diffusers and SAT (SwissArmyTransformer). The Diffusers version supports memory optimizations including sequential CPU offload, VAE slicing, and VAE tiling, enabling inference on consumer GPUs. CogVideoX-2B can run on GTX 1080 Ti, while CogVideoX-5B runs on RTX 3060 class GPUs. Quantized inference via TorchAO supports INT8 precision, further reducing memory requirements. The repository provides Colab notebooks for T2V, I2V, and V2V tasks on free T4 instances. Fine-tuning is supported through LoRA training with reduced GPU memory requirements, and the CogKit toolkit provides a unified framework for fine-tuning and inference across CogView4 and CogVideoX series. A community-developed fine-tuning framework (cogvideox-factory) enables CogVideoX-5B fine-tuning on a single 4090 GPU. The repository also includes DDIM inversion support, a prompt optimization guide using large language models, and the CogVLM2-Caption model used for converting video data into text descriptions during training. The original CogVideo model (ICLR 2023) is available in a separate branch as the first open-source Transformer-based text-to-video generation model. CogVideoX-2B is released under Apache 2.0 License.