clickstream-pipeline

This project provides a complete serverless clickstream data pipeline built on AWS services. It accepts real-time click events through Amazon API Gateway, routes them into Kinesis Data Streams for ingestion, transforms and partitions the data into Parquet format via Kinesis Firehose, and stores everything in S3 as a data lake. AWS Glue catalogs the schema while Athena enables ad-hoc SQL queries. An EventBridge-scheduled Lambda performs automatic partition repair. The repository includes Infrastructure as Code using AWS SAM and CloudFormation for repeatable, version-controlled deployments. Incoming payloads use Base64-encoded JSONL format with deviceId serving as the Kinesis partition key. Transformed data is stored with year/month/day partitioning and Snappy compression for cost-effective querying. Production considerations are documented extensively, covering ingestion rate optimization, client-side compression, schema evolution handling, monitoring with CloudWatch, security patterns, and alternatives to batch partition repair for real-time scenarios. The original data is sourced from a Kaggle clickstream dataset.