About this project

This repository demonstrates how to build a low‑latency analytics solution for web traffic on Google Cloud Platform. The architecture consists of four main parts: 1. **Event ingestion** – A Pub/Sub topic receives click‑stream messages generated by a simple Python event generator. Pub/Sub provides a durable, globally‑available message bus that decouples producers from consumers. 2. **Streaming processing** – An Apache Beam pipeline runs on Dataflow (runner=DataflowRunner) in streaming mode. The pipeline reads Pub/Sub events, extracts relevant fields and updates metric counters in Cloud Memorystore (Redis). Redis is used as an in‑memory store because its native set and HyperLogLog commands (`SCARD`, `SINTERSTORE`, `PFADD`, `PFCOUNT`) make unique‑visitor counting and set‑based analytics very efficient. 3. **Metrics store** – Cloud Memorystore (Redis) holds the real‑time metrics. By leveraging Redis' built‑in data structures, the solution avoids custom aggregation code and achieves sub‑second query latency. 4. **Visualization** – A Spring Boot application (located in the `dashboard/` folder) connects to Redis via the Jedis client, reads the aggregated values and renders them with Google Charts. The UI offers auto‑refresh, allowing marketers or operators to monitor KPIs such as active users, experiment participation, conversion rates, and user overlap. The repository includes scripts and instructions to: - Set up required GCP services (Pub/Sub, Dataflow, Memorystore, Cloud Storage, VPC networking). - Deploy the Dataflow pipeline using Maven. - Run the dummy event generator in a Python virtual environment. - Launch the Spring Boot dashboard behind a proxy VM for VPC‑restricted Redis access. - View the live dashboard via Cloud Shell web preview. All components are fully managed services, so the solution scales automatically and requires minimal operational overhead. It serves as a starter kit for anyone needing real‑time web analytics, A/B‑test monitoring, or any click‑stream based KPI reporting on GCP.