mediacrawler

MediaCrawler is an open-source multi-platform social media data collection tool that supports scraping public data from major platforms including Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. Built on the Playwright browser automation framework, it retrieves data by maintaining authenticated browser sessions, eliminating the need to reverse-engineer complex encryption algorithms. Features include keyword-based search, crawling by specific post IDs, extraction of secondary-level comments, scraping of creator profile pages, login state caching, integrated IP proxy pool, and generation of comment word clouds. Collected data can be saved in multiple formats: CSV, JSON, JSONL, Excel, SQLite, and MySQL. The tool provides both a command-line interface and a WebUI for visualization. Users can configure crawler parameters via the WebUI, monitor real-time operation status and logs, and preview or export data directly. To use MediaCrawler, users must enable Chrome's remote debugging feature to reuse existing login sessions, reducing the risk of platform detection and blocking. The project explicitly states that it is intended solely for learning and research purposes, and commercial or illegal use is strictly prohibited.