pathway

Pathway is a live data framework for Python that handles both streaming and batch data processing through a single API. It is built on a Rust engine using Differential Dataflow for incremental computation, allowing pipelines to be updated in real time as data arrives. The framework provides connectors for Kafka, PostgreSQL, Google Drive, SharePoint, and Airbyte (300+ sources), along with custom connector support via Python. For AI pipelines, Pathway includes an LLM xpack with wrappers for major LLM services, embedders, splitters, parsers, and an in-memory real-time vector index. It integrates with LangChain and LlamaIndex for RAG applications. Stateful transformations include joins, windowing, and sorting, with support for any Python function or library. Persistence allows pipelines to restart after crashes or updates. Consistency is at-least-once in the free version and exactly-once in the enterprise tier. Deployment options include local execution via `pw.run()`, Docker, and Kubernetes. A monitoring dashboard shows message throughput and latency per connector. The engine supports multithreading, multiprocessing, and distributed computation out of the box.