About this project

This project began as a single Selenium scraper for Eurojackpot and grew into a small data pipeline. It collects lottery draw data on a schedule, stores it in SQLite, and uses it to forecast OPAP SA's quarterly Gross Gaming Revenue (GGR), then backtests each forecast against figures OPAP actually reports. The author describes it as an educational/hobby project: the model is a simple regression trained on very few data points, and it is not investment advice. Architecture and components: - Scrapers: eurojackpot_api.py uses a reverse-engineered JSON API behind eurojackpot.com's Angular frontend (wlinfo/WL_InfoService), returning stakes, full prize-tier breakdowns, winning numbers for any date, and draw dates back to 2012. eurojackpot_playwright.py is a fallback if that endpoint changes. joker_scraper.py and lotto_scraper.py use Playwright against allwyn.gr, which returns 403 to plain HTTP clients and offers no usable API; _allwyn_common.py holds shared page-parsing logic. opap_ir_scraper.py is a best-effort scrape of OPAP investor-relations press releases. - Database: db/schema.sql defines the SQLite schema; db/repository.py is the sqlite3 data-access layer. - Pipeline: scrape_latest.py runs on a schedule via GitHub Actions; backfill_history.py backfills Eurojackpot draws to 2012; build_quarterly_features.py aggregates draws into per-reporting-period features; load_manual_ggr.py loads data/manual/opap_quarterly_ggr.csv. - Forecasting: dataset.py joins features to actual GGR; train_model.py trains Ridge/GradientBoosting regression with cross-validation; predict.py forecasts the next not-yet-reported period; backtest.py performs a walk-forward backtest. Data sources and limitations: Eurojackpot has a real JSON API and is the primary source. Joker/Lotto results pages publish total columns played (a genuine stakes figure at €1/column) plus prize-tier tables, but only the current draw is scraped; historical backfill would require driving the site's search form, which is not implemented. OPAP's reported GGR comes from investors.opap.gr, but financial PDF layouts vary, so opap_ir_scraper.py is best-effort and not the source of truth. Instead, data/manual/opap_quarterly_ggr.csv is hand-populated from real OPAP press releases, with each row citing its source URL, and is what the backtest grades against. Forecasting and backtesting: the model predicts OPAP's numerical_lotteries segment GGR from aggregated Eurojackpot/Joker/Lotto stakes and payouts. backtest.py walks forward through history, training only on strictly earlier periods, predicting, and comparing, writing data/backtest_report.csv with per-period MAPE and whether it called growth versus decline correctly. It needs at least two populated periods in the manual CSV and becomes more meaningful as more reports are added. Running locally: install requirements and Playwright's Chromium, then run the pipeline modules in order (scrape_latest, backfill_history, load_manual_ggr, build_quarterly_features, train_model, backtest, predict). CI/CD: scrape.yml runs scrape_latest daily and commits the updated DB; backtest.yml runs roughly quarterly and on demand via workflow_dispatch, loading the manual CSV, building features, retraining, backtesting, predicting the next period, and committing results. The README notes GGR composition by segment (lotteries, sports betting, VLTs, online casino, instants/passive) and includes an original disclaimer that the project does not calculate exact real-world profit, since taxes, marketing and operating costs are out of scope.