realtime-ecommerce-analytics-flink

RealTime-ECommerce-Analytics (RTEA) is a high-performance streaming pipeline written in Python that captures e-commerce clickstream data from a REST API and routes it through Confluent Cloud Kafka into ClickHouse for real-time analytical querying. ## Architecture The pipeline consists of four layers: **Data Source** — A simulated clickstream generator API hosted on Railway.app producing approximately 10,000 messages per second in JSON format. **Message Broker** — Confluent Cloud Kafka with a topic `clickstream-events` across 6 partitions, SASL_SSL authentication, Snappy compression, and 7-day retention. **Processing Layer** — Producer service using `kafka-python` with optimized settings (16 KB batches, gzip compression, 32 MB buffer, rate-limited to 10K msg/sec) and consumer service with batch processing (1000 events per batch, auto-commit offsets, field transformation). **Analytics Storage** — ClickHouse Cloud with a MergeTree engine table ordered by `(timestamp, user_id)`, LZ4 compression, and a 30-day TTL. ## Data Flow 1. The producer polls the external API every second, collects messages into a queue, and publishes them in batches to the Kafka topic. 2. Multiple consumer instances each subscribe to assigned partitions, deserialize JSON, transform timestamps and fields, and write batches to ClickHouse. 3. Users query the resulting `ecommerce.clickstream_events` table via SQL. ## Key Configuration ### Producer - Batch size: 16 KB - Compression: gzip - Buffer memory: 32 MB - Retries: 3 ### Consumer - Batch size: 1000 events - Batch timeout: 5 seconds - Offset auto-commit enabled ### ClickHouse Table ```sql CREATE TABLE clickstream_events ( user_id String, session_id String, timestamp UInt64, event_type String, product_id String, product_category String, price Float64, quantity Int32, source String, received_time DateTime DEFAULT now() ) ENGINE = MergeTree() ORDER BY (timestamp, user_id) ``` ## Reported Performance - Sustained throughput: 650+ events/second - Total events processed: 510,414+ - Sub-second end-to-end latency ## Quick Start 1. Install dependencies with `uv sync` 2. Copy `.env.example` to `.env` and fill in Confluent Cloud and ClickHouse credentials 3. Run the producer: `uv run python services/producer/src/simple_producer.py` 4. Run the consumer: `uv run python services/consumer/src/simple_consumer.py` 5. Verify data: `uv run python check_clickhouse.py` ## Troubleshooting The repository includes diagnostic scripts (`debug_kafka.py`, `test_clickhouse_insert.py`) and covers common failure modes including API connection timeouts, Kafka consumer lag, message deserialization errors, and ClickHouse insert/schema mismatches.