vibevoice

VibeVoice is an open-source voice AI project from Microsoft comprising both Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) models. A core technical feature is its use of continuous speech tokenizers operating at an ultra-low frame rate of 7.5 Hz to preserve audio fidelity while improving computational efficiency for long sequences, combined with a next-token diffusion framework. Key models and capabilities documented in the repository: 1. VibeVoice-ASR: A unified speech-to-text model that processes up to 60 minutes of continuous audio in a single pass. It produces structured transcriptions containing speaker identity (Who), timestamps (When), and content (What). It supports user-customized hotwords for domain-specific terms and is natively multilingual with support for over 50 languages. A streaming variant transcribes audio as it arrives. It is integrated into Hugging Face Transformers and supports vLLM inference. 2. VibeVoice-ASR-BitNet: An edge CPU inference engine using heterogeneous quantization (I8_S + I2_S), compressing the model from 4.62 GB to 1.58 GB for real-time inference on CPU threads without a GPU. 3. VibeVoice-TTS: A long-form multi-speaker text-to-speech model capable of synthesizing up to 90 minutes of speech in a single pass with up to 4 distinct speakers. It supports expressive speech and multiple languages including English and Chinese. Note: The TTS code was removed from the repository due to responsible AI concerns after instances of misuse. 4. VibeVoice-Realtime-0.5B: A lightweight (0.5B parameter) real-time streaming TTS model supporting streaming text input and robust long-form speech generation, with experimental multilingual voices. The repository includes documentation, finetuning code for ASR, demos, and links to model weights on Hugging Face. The project states it is intended for research and development purposes only and not recommended for commercial or real-world applications without further testing.