crawlee-python

Crawlee is a comprehensive web scraping and browser automation library for Python, designed to help developers build reliable crawlers quickly. It provides an end-to-end solution for crawling and scraping, from URL discovery to data extraction and storage. Key features include: - **Unified interface** for both HTTP-based and headless browser crawling - **Automatic parallel crawling** based on system resources - **Type hints** for better developer experience and reduced bugs - **Automatic retries** on errors or blocking - **Proxy rotation** and session management - **Configurable request routing** to direct URLs to appropriate handlers - **Persistent URL queue** for crawling - **Pluggable storage** for tabular data and files - **Robust error handling** Crawlee offers two main crawler types: - **BeautifulSoupCrawler**: Uses HTTP library and BeautifulSoup for HTML parsing, ideal for efficient data extraction without JavaScript execution - **PlaywrightCrawler**: Uses headless browser via Playwright, suitable for JavaScript-heavy sites requiring interaction It supports various parsing libraries including Parsel, BeautifulSoup, and raw HTTP, and can download files like HTML, PDF, JPG, PNG. The library works in both headful and headless modes. Compared to Scrapy, Crawlee is asyncio-based, has complete type hints, simpler integration (regular Python scripts), state persistence during interruptions, and organized data storages (datasets and key-value stores). Installation is straightforward via pip with optional extras for additional features. A CLI tool is available for quick project setup with templates. Crawlee is developed by Apify and can be deployed on the Apify platform for cloud execution. Documentation, guides, and examples are available on the Crawlee project website, with a TypeScript implementation also available for JavaScript/TypeScript projects.