top_news

Top News URLs is a data collection project that has been gathering news article URLs hourly from fifteen major news sources since 2022. The repository stores one URL per line in JSON arrays, making collection diffs easy to review. Sources include ABC, BBC, CBS, CNN, Deutsche Welle, The Guardian, Los Angeles Times, NBC, NPR, The New York Times, Politico, ProPublica, USA Today, The Washington Post, and Al Jazeera. The project provides a command-line tool built with Python 3.12+ and uv. Users can run `top-news update` to collect URLs from all sources or a specific site, `top-news to-parquet` to export the data as Parquet files with `site` and `url` columns, and `top-news upload` to add files to Dataverse using an API token. Historical full-text database releases are available through Dataverse with a CC0 license, while the URL collection in the repository is CC BY 4.0 and the code is MIT licensed. Collection methods evolved over time: from 2022 to September 2026 the project used eleven independent RSS scripts, and from September 2026 onward it uses a shared collector with RSS feeds plus news sitemaps for CNN and USA Today. The README documents known coverage gaps, including late-starting sources like BBC, The Guardian, Al Jazeera, and Deutsche Welle, as well as feed interruptions for CNN and USA Today. Development checks include Ruff, formatting, pytest, and pre-commit, with CI running in standard Python 3.12 and 3.14 Docker images.