daft

Daft is a high-performance data engine designed for AI and multimodal workloads. It lets users process images, audio, video, embeddings and structured data within a single framework, and is described as Python-native with a Rust-powered core. Key capabilities highlighted in the README: - Native multimodal processing: handle images, audio, video and embeddings alongside structured data in one framework. - Built-in AI operations: run LLM prompts, generate embeddings and classify data at scale using OpenAI, Transformers or custom models. - Python-native, Rust-powered: Python at its core with Rust under the hood, avoiding JVM complexity. - Seamless scaling: start locally and scale to distributed clusters on Ray or Kubernetes. - Universal connectivity: access data from S3, GCS, Iceberg, Delta Lake, Hugging Face and Unity Catalog. - Out-of-box reliability: intelligent memory management and sensible defaults. Installation is via `pip install daft` and requires Python 3.10 or higher. The README points to a quickstart that loads a real-world e-commerce dataset, processes product images and runs AI inference at scale, plus examples, a user guide and an API reference. A benchmarking page is also referenced. The project welcomes contributions and lists good first issues. It collects non-identifiable telemetry via Scarf, which can be disabled by setting `DO_NOT_TRACK=true`; the README states no session IDs or user identifiers are collected and no proprietary code or data is gathered. A comparison table in the README positions Daft alongside Pandas, Polars, Modin, Ray Data, PySpark and Dask DF across attributes such as query optimizer, multimodal support, distributed execution, Arrow backing, vectorized execution and out-of-core processing. Daft is licensed under Apache 2.0.