sensevoice

SenseVoice is a speech foundation model with multiple speech understanding capabilities: automatic speech recognition (ASR), spoken language identification (LID), speech emotion recognition (SER), and audio event detection (AED). Scope of the released checkpoint: SenseVoiceSmall supports ASR and language ID for Mandarin, Cantonese, English, Japanese, and Korean, together with emotion and audio-event tags. The broader SenseVoice research work reports training on more than 400,000 hours and support for more than 50 languages, but the released small checkpoint covers the five languages above. Speaker diarization is not an output of the checkpoint itself; it is a composed FunASR pipeline using separate FSMN-VAD and CAM++ models. Highlights - Rich transcription: emotion recognition tags and sound event detection for events such as background music, applause, laughter, crying, coughing, and sneezing. - Efficient inference: a non-autoregressive end-to-end framework for low-latency inference. In the project's benchmark setup, SenseVoiceSmall is reported to run more than 5x faster than Whisper-Small and 15x faster than Whisper-Large at a similar parameter count. - Finetuning scripts and strategies for adapting to long-tail data. - Service deployment pipeline supporting multi-concurrent requests, with client-side languages including Python, C++, HTML, Java, and C#. Usage Install dependencies with pip install -r requirements.txt. Examples and the composed diarization path require funasr>=1.3.26 (the current deployment path recommends funasr==1.4.14). Basic inference uses FunASR's AutoModel with the iic/SenseVoiceSmall model, optional FSMN-VAD segmentation for long audio, language selection (auto, zh, en, yue, ja, ko, nospeech), inverse text normalization, dynamic batching, and VAD merging. A rich_transcription_postprocess helper formats the raw output. Long audio without VAD: passing an hour-long waveform to a single generate call can grow encoder memory far beyond the audio size. The long_audio_no_vad.py script decodes through ffmpeg and runs SenseVoice on fixed 30-second windows with 2 seconds of overlap, keeping raw chunk outputs in a JSONL file. Window offsets describe input boundaries, not word timestamps, and fixed boundaries can still affect recognition around a cut. Speaker diarization: composed through FunASR with FSMN-VAD, CAM++ speaker labels, and a punctuation model. Speaker labels are anonymous clusters, not recognized personal identities. With output_timestamp=True, timestamps are [start_ms, end_ms] pairs aligned one-to-one with words. Export and edge deployment: ONNX and Libtorch export paths are documented, and a llama.cpp/GGUF runtime allows running SenseVoice as a self-contained binary with built-in FSMN-VAD and no Python at runtime, aimed at CPU and edge devices. Service and containers: a FastAPI deployment listens on port 50000, and Docker/Docker Compose files support CPU and GPU runs with a model cache volume. Finetuning: data preparation uses JSONL records with key, source, target, language, emotion, and event labels; helper commands convert wav.scp and text files into train/val JSONL. Emotion labels include HAPPY, SAD, ANGRY, NEUTRAL, FEARFUL, DISGUSTED, and SURPRISED; event labels include BGM, Speech, Applause, Laughter, Cry, Sneeze, Breath, and Cough. A finetune.sh script and a webui.py demo are provided. Benchmarks: the project compares multilingual ASR against Whisper on AISHELL-1, AISHELL-2, Wenetspeech, LibriSpeech, and Common Voice, reporting advantages for Chinese and Cantonese recognition. For emotion recognition it reports competitive zero-shot results on Chinese and English test sets, with a reproducible SER evaluation contract. For audio event detection it compares against BEATS and PANN on ESC-50, noting gaps versus specialized AED models. Ecosystem: SenseVoice is part of the FunAudioLLM family alongside FunASR, Fun-ASR, and CosyVoice. Third-party integrations include Triton/TensorRT deployment, sherpa-onnx (supporting many programming languages and platforms such as iOS, Android, and Raspberry Pi), SenseVoice.cpp, streaming-sensevoice, OmniSenseVoice, and hotword-enhanced variants.