speech

NVIDIA NeMo Speech is an open-source framework aimed at researchers and PyTorch developers who work on speech models. According to its README, it targets Automatic Speech Recognition (ASR), Text-to-Speech (TTS) and Speech LLMs, and is intended to help users create, customize and deploy new models by reusing existing code and pre-trained checkpoints. Scope and positioning - The repository states that it has pivoted to focus on audio, speech and multimodal LLMs; the final pre-split NeMo release covering additional modalities is v2.7.3. - The current line is NeMo Speech 3.0, published as release v3.0.0 and shipped in an NGC container. - Model checkpoints and demos are collected in a HuggingFace collection referenced from the README. Documentation and releases - Developer documentation is hosted for the latest release (3.0.0) and for the nightly main branch. - The README lists dated updates describing model releases such as multilingual TTS checkpoints, streaming ASR models, unified offline/streaming ASR, and speech recognition/translation models for European languages. These are release notes; no independent benchmark verification is provided here. Requirements - Python 3.12 or above, PyTorch 2.7 or above (CPU or CUDA), and an NVIDIA GPU with CUDA for training (recommended for inference). - The project says it installs on top of an existing Python/PyTorch/CUDA stack without replacing it; the versions pinned in uv.lock and the official container (Python 3.13, PyTorch 2.11 with CUDA 12.9 or PyTorch 2.12 with CUDA 13.2) are described as actively tested combinations rather than hard requirements. - A note warns that since PyTorch 2.6 torch.load defaults to weights_only=True, and some checkpoints may need TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1, which should only be used with trusted files because loading untrusted files can risk arbitrary code execution. Installation options - Recommended: install from source with uv, syncing extras such as all and cu13 (or cu12), which reproduces the tested stack in a .venv; optional groups add tests or docs. - Docker: pull the prebuilt NGC container or build from the provided Dockerfile, with GPU_TARGET options for H100+ or A100. - pip fallback: install PyTorch first, then nemo-toolkit with asr and tts extras; the README notes that uv sync --locked should not be used for a bring-your-own stack because it applies the lockfile and replaces the existing Python/PyTorch/CUDA. - Optional accelerated backends (Transformer Engine, FlashAttention, Mamba, grouped-GEMM/MoE, DeepEP) are described as source-built kernels from compiled extras, with the Automodel backend able to run without compiled dependencies. Licensing and contribution - Licensed under Apache License 2.0. - Community contributions are welcomed through the CONTRIBUTING.md process. This overview reflects only what the repository README states; it does not verify accuracy, performance or ranking claims made in release notes.