spate

Spate is a streaming ETL framework for Rust that provides at-least-once delivery semantics. It is designed as an alternative to general-purpose stream processors and hand-written consumer loops: transformations are ordinary Rust functions compiled directly into the pipeline, while delivery guarantees, backpressure, checkpointing, rebalancing, and drain-on-shutdown are handled by the framework. A pipeline runs in four stages. Extract uses one consumer per process and fans partitions across CPU-pinned threads as zero-copy lanes. Transform chains stateful closures into a single loop over borrowed records with no per-record allocation. Load shards and replicates sinks asynchronously, using bounded per-shard queues as the backpressure signal. Observe advances the source watermark only behind data acknowledged as durable by the sink, so commits trail delivery. Connectors are provided as optional feature-gated crates: Kafka source and sink, ClickHouse sink with deduplication tokens and replica rotation, coordinated S3 backfill source, Avro and JSON deserialization, multi-instance work assignment, and a synthetic datagen source for demos and tests. Nothing is enabled by default, so unused connector code is not compiled. The project includes a user guide, API reference, numbered invariants, architectural decision records, metrics documentation, and Docker/Kubernetes examples. Testing covers proptest-based property tests, loom concurrency modeling, librdkafka MockCluster runs, and container-based integration tests. The project is under active initial development with unstable 0.x APIs and is licensed under Apache-2.0.