imagebind

ImageBind is FAIR Meta AI’s PyTorch implementation of a multimodal embedding model. It learns a joint representation for six modalities: images, text, audio, depth, thermal imagery, and IMU data. Once encoded into the shared space, embeddings from different modalities can be compared, enabling cross-modal retrieval and similarity analysis. The accompanying research also describes emergent uses such as combining modalities through arithmetic, cross-modal detection, generation, and zero-shot classification. The repository includes the imagebind_huge pretrained checkpoint, model code, data-loading and transformation utilities, and an example that extracts image, text, and audio embeddings and compares them using matrix multiplication and softmax scores. It reports zero-shot results on ImageNet-1K, Kinetics-400, NYU Depth, ESC, LLVIP, and Ego4D benchmarks. Installation requires PyTorch 2.0 or newer and Python 3.10 is used in the provided Conda example. Windows users may additionally need soundfile for audio I/O. The code and model weights are released under the CC-BY-NC 4.0 license, which restricts commercial use. The work was presented at CVPR 2023 as a highlighted paper.