cog

Cog is an open-source tool that packages machine learning models into standard, production-ready Docker containers. It targets the common difficulty of shipping ML models to production by removing the need to hand-write Dockerfiles and manage dependency compatibility. Key capabilities described in the README: - Docker image generation from a simple cog.yaml configuration file, applying best practices such as Nvidia base images, efficient dependency caching, specific Python version installation, and sensible environment variable defaults. - Automatic handling of CUDA, cuDNN, PyTorch, TensorFlow, and Python version compatibility, avoiding manual version matrix management. - Model input and output definition using standard Python types via the BaseRunner interface (setup and run methods), with Cog generating an OpenAPI schema and validating inputs and outputs. - Automatic HTTP inference server: model types dynamically generate a RESTful HTTP API served by a high-performance Rust/Axum server. - CLI commands: cog run (run a model with inputs), cog build (build a deployable Docker image), and cog serve (combine build and serve on a local port). - Deployment flexibility: built Docker images run anywhere Docker images run, including self-hosted infrastructure or Replicate. - Platform support: macOS, Linux, and Windows 11 via WSL 2. Requires Docker (and Buildx when using Docker Engine instead of Docker Desktop). - Installation via Homebrew, an install script, manual binary download, or inside a Docker image. The tool is positioned as a way for researchers to ship models to production without deep Docker or infrastructure engineering knowledge, and is created by the team behind Replicate.