colly

Colly is a lightweight and high-performance web scraping and crawling framework written in Go. It provides a clean, intuitive API for building scrapers, crawlers, and spiders, making it easy to extract structured data from websites for applications like data mining, processing, or archiving. Key features include: - Clean and elegant API - High speed (over 1k requests per second on a single core) - Automatic handling of request delays and maximum concurrency per domain - Automatic cookie and session management - Support for synchronous, asynchronous, and parallel scraping - Built-in caching - Automatic encoding of non-Unicode responses - Robots.txt support - Distributed scraping capabilities - Configuration via environment variables - Extensible through add-ons The README includes a simple example demonstrating how to create a collector, visit all links on a page, and handle requests. It also lists several open-source projects that use Colly, such as a search engine, a Steam game database, and a website cloner. The project is actively maintained and accepts contributions.