nvsentinel

NVSentinel is an NVIDIA open-source project for GPU infrastructure reliability on Kubernetes. It addresses the problem of a single faulty GPU silently corrupting training runs or leaving nodes idle: the system detects faults as they occur, protects workloads by cordoning and draining the affected node, and remediates it via a targeted GPU reset or full reboot, returning it to service without manual paging. Core capabilities: - Detect: real-time GPU, NIC and system-level fault detection through DCGM, syslog/journalctl, cloud provider maintenance events, Kubernetes object monitoring and pattern-based health event analysis. - Protect: cordon the faulty node so the scheduler stops placing new work on it, and optionally drain existing workloads. Cordon-only mode is possible by omitting the node drainer. - Remediate: automatic repair through a privileged on-node job (GPU reset or reboot), with optional cloud provider API integration for reboots. - Preflight (optional): an init container that verifies node readiness before a job starts, using Kubernetes gang scheduling; enabled per namespace via a label. Architecture: independent modules coordinated through a shared MongoDB event store and the Kubernetes API, with no direct module-to-module communication. Health monitors (GPU, syslog, CSP, NIC, health events analyzer, Kubernetes object monitor) send events over gRPC to platform connectors, which persist them and update node conditions and events. Fault management components (fault quarantine, node drainer, fault remediation, janitor) reconcile changes from the store and act on the Kubernetes API. The design is extensible with pluggable health monitors, drain strategies and remediation actions. Prerequisites: Kubernetes 1.34+, Helm 3.0+, NVIDIA GPU Operator with DCGM standalone mode enabled (dcgm.enabled=true), cert-manager v1.21+, and persistent storage for a database. Installation and adoption: a single Helm command installs or upgrades NVSentinel from an OCI chart. By default only health monitoring is enabled, which is non-disruptive. Users can progressively enable cordon, drain, remediation, janitor and preflight through Helm values. A local fault injection demo runs in a KIND cluster without GPU hardware, using simulated DCGM to demonstrate detection, quarantine and cordon. Supported hardware: validated on NVIDIA Volta, Ampere, Hopper, Ada Lovelace and Blackwell architectures. Distribution is exclusively via the OCI Helm chart, GHCR container images, GitHub Releases and the official documentation site; container images carry Sigstore-signed CycloneDX SBOM and SLSA build provenance attestations. The project is licensed under Apache 2.0 and is described as ready for production testing, with APIs and configurations subject to change between releases.