sam2

SAM 2 (Segment Anything Model 2) is a foundation model from Meta AI / FAIR for promptable visual segmentation in both images and videos. It extends the original SAM approach by treating an image as a single-frame video, using a transformer architecture with streaming memory for real-time video processing. The repository provides the official inference code, downloadable trained checkpoints, example notebooks, training/fine-tuning code, and a locally deployable web demo. Key capabilities described in the README: - Image prediction: the SAM2ImagePredictor class offers an interface similar to SAM for prompt-based segmentation on static images, including automatic mask generation. - Video prediction: a video predictor supports adding point or box prompts, refining them, and propagating masklets across a video, with multi-object tracking and per-object inference state. - Model checkpoints: SAM 2.1 checkpoints (tiny, small, base_plus, large) released September 2024, plus the earlier SAM 2 checkpoints from July 2024, with reported size, speed and benchmark figures on SA-V, MOSE and LVOS v2. - Hugging Face loading: models can be loaded via from_pretrained for both image and video predictors. - Training and fine-tuning: code is provided for training or fine-tuning on custom image, video, or mixed datasets. - Web demo: frontend and backend code for a locally deployable demo similar to the hosted SAM 2 demo. - SA-V dataset: the largest video segmentation dataset to date, built with a model-in-the-loop data engine; dataset documentation is included. Installation requires Python 3.10+, PyTorch 2.5.1+ and TorchVision 0.20.1+, with a CUDA toolkit for compiling the custom kernel; Windows users are advised to use WSL. Optional extras install Jupyter and matplotlib for the notebooks. The README notes that a failed CUDA extension build can be ignored, with only limited post-processing impact. A December 2024 update added full model compilation for video object segmentation speedup and an updated SAM2VideoPredictor supporting independent per-object inference and adding new objects after tracking starts. Licensing: model checkpoints, demo code and training code are under Apache 2.0, with the Inter Font and Noto Color Emoji under the SIL Open Font License 1.1. Third-party GPU connected-component code is adapted from cc_torch with its own license file.