yue

YuE2 is a music generation project from the multimodal-art-projection group (HKUST, M·A·P, Tokenwave.AI, NYU, Stanford, MBZUAI, NOIZ, ACE Studio). Its stated goal is to unify symbolic and audio music generation: given lyrics and a style prompt, it first writes a melody-and-chord plan, then realizes that plan as a complete song with vocals and accompaniment. Key capabilities described in the README: - Symbolic planning as a white-box interface. The generated composition is an editable score (ABC notation) that a person or an agent can read, play, inspect and modify before rendering. Melody and chords become explicit controls. - Zero-shot covers. A source recording can be transcribed with the companion SheetSage2 model, and the resulting melody score can be re-rendered in a new style. The README recommends cot="melody" with a score lacking chord symbols so the accompaniment adapts to the new style. - Agentic music editing. A plan can be exported, revised (harmony, melody, tempo, form, lyrics), and re-rendered as a new complete recording. The README notes that editing generates a new recording and does not preserve the original waveform outside an edit. - An agent skill package (skills/yue2-music/SKILL.md) that documents how an agent can generate songs, transcribe and cover recordings, edit ABC scores, check musical invariants, and organize listening comparisons. Architecture and pipeline: a single AR–NAR Mixture-of-Transformers backbone predicts score and semantic tokens autoregressively, then generates acoustic latents with flow matching; a VAE decodes latents into 48 kHz stereo audio. The staged Python API exposes plan() → generate_semantic() → synthesize() → decode(). Generation modes include cot="full" (editable melody-and-chord plan, default), cot="melody" (melody plan with free accompaniment, recommended for covers), cot="off" (direct from lyrics and style), and abc=... to supply your own score. Requirements and quick start: Linux, Python 3.12, an NVIDIA GPU with BF16 support and 24 GB VRAM. Installation is via pip install . from a clone of the repository, followed by examples/generate.py. Model files download from Hugging Face on first use. Outputs retain the score, semantic tokens, acoustic latents, generation settings and model identities alongside the audio. Companion models and resources: YuE2-3B (generation, planning, covering, editing), YuE2-Vae and YuE2-Vae-legacy (decoders), SheetSage2 (audio-to-score transcription), MERT-v2-FullSong and MERT-v2-30s (music representations), and the WildSongBench evaluation dataset. MERT2 feature extraction is optional for generation. The README reports benchmark results on WildSongBench (192 prompts, automatic evaluation) comparing YuE2 with several proprietary and open systems, and reports zero-shot cover results on 948 works. It also reports MERT2 results on MARBLE and GTZAN, and SheetSage2 results on transcription benchmarks. These are the project's own reported figures; the README itself notes that rankings vary by metric and that small gaps between top means do not establish statistical significance. Licensing: first-party code, agent skill and documentation are Apache 2.0; model weights are separately licensed CC BY-NC 4.0; third-party components retain their original licenses. The earlier YuE-v1 code, documentation and license are preserved on a separate branch.