hunyuanvideo

HunyuanVideo is an open-source video foundation model developed by Tencent, designed for large-scale video generation. The repository provides PyTorch model definitions, pre-trained weights, and inference/sampling code. Key technical features include: - Unified Image and Video Generative Architecture: Uses a "Dual-stream to Single-stream" hybrid Transformer design with full attention mechanism, processing video and text tokens independently before fusing them for multimodal information integration. - MLLM Text Encoder: Utilizes a pre-trained Multimodal Large Language Model with a Decoder-Only structure as the text encoder, offering better image-text alignment and complex reasoning compared to CLIP or T5-XXL. - 3D VAE: Employs a Causal 3D VAE with CausalConv3D to compress pixel-space videos into a compact latent space with compression ratios of 4x for video length, 8x for space, and 16x for channels. - Prompt Rewrite: Includes a fine-tuned prompt rewrite model (based on Hunyuan-Large) with Normal and Master modes to adapt user prompts for better video generation results. The model has over 13 billion parameters. The repository supports single-GPU inference, multi-GPU parallel inference via xDiT, and FP8 quantized weights for reduced GPU memory usage. It requires a minimum of 45GB GPU memory (for 544x960x129f) or 60GB (for 720x1280x129f), with 80GB recommended. The project has spawned multiple derivatives including HunyuanVideo-I2V (image-to-video), HunyuanVideo-Avatar (audio-driven animation), and HunyuanCustom (customized video generation). It integrates with Diffusers and ComfyUI, with community contributions offering quantized versions, acceleration tools, and various optimizations.