streaming-pipeline-kafka-spark

This project provides a fully Dockerized, local real-time streaming analytics pipeline that demonstrates Spark Structured Streaming concepts without requiring any cloud accounts. ## What It Does A Python producer simulates clickstream events sent to a Redpanda topic (Kafka-compatible), and a PySpark Structured Streaming job consumes those events, applies 1-minute tumbling window aggregations with watermarking for late events, and outputs results to the console or Parquet files. ## Architecture - **Python Producer**: Generates simulated clickstream JSON events containing URL and timestamp data - **Redpanda**: Kafka-compatible broker running in Docker, no Zookeeper required - **PySpark Structured Streaming**: Reads from Kafka as an unbounded DataFrame, parses JSON with an explicit schema, applies event-time windowing with watermarks, and writes aggregated page-view counts per URL per window ## Key Concepts Demonstrated - Reading a Kafka topic as an unbounded DataFrame with Spark Structured Streaming - Parsing JSON event payloads with schema-on-read - Event-time windowing (1-minute tumbling windows) with watermarking to handle late-arriving events - Continuous micro-batch aggregation of page views per URL - Output to console for learning, with Parquet sink option for persistence ## Project Structure ``` streaming-pipeline-kafka-spark/ ├── docker-compose.yml # Redpanda + Spark containers ├── producer/ │ └── click_event_producer.py # simulated clickstream producer ├── spark_app/ │ └── streaming_aggregation.py # Structured Streaming job └── requirements.txt ``` ## Getting Started ```bash git clone https://github.com/Kornelius99/streaming-pipeline-kafka-spark.git cd streaming-pipeline-kafka-spark docker-compose up -d # Terminal 1: start producing events python producer/click_event_producer.py # Terminal 2: run the Spark Streaming job docker-compose exec spark spark-submit \ --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1 \ /opt/spark_app/streaming_aggregation.py ``` Results appear as 1-minute windowed page-view counts printed to the console, updating as new events arrive. ## Why Redpanda Redpanda speaks the Kafka protocol so existing Kafka client code works unmodified, but it ships as a single container with no Zookeeper dependency, keeping the docker-compose simple for learning purposes. ## License MIT