clip

CLIP (Contrastive Language-Image Pre-Training) is a neural network model developed by OpenAI, trained on a diverse set of (image, text) pairs. It can predict the most relevant text snippet for a given image using natural language instructions, without task-specific optimization, similar to the zero-shot capabilities of GPT-2 and GPT-3. The model achieves performance comparable to the original ResNet50 on ImageNet 'zero-shot' without using any of the original 1.28M labeled examples. The repository provides the official implementation, including model loading, tokenization, and feature encoding functions. It supports multiple pre-trained models (e.g., ViT-B/32) and offers examples for zero-shot prediction and linear-probe evaluation. The code is designed for use with PyTorch and torchvision, and can run on CUDA GPUs or CPU. Key API methods include `clip.available_models()`, `clip.load()`, and `clip.tokenize()`. The loaded model supports `encode_image()`, `encode_text()`, and a forward method that returns logit scores based on cosine similarity between image and text features. The README includes installation instructions, usage examples, and links to related projects such as OpenCLIP and Hugging Face's implementation.