crawl4ai

Crawl4AI is an open-source Python library and CLI tool for web crawling and scraping, designed to produce clean, LLM-ready Markdown output. It is widely used for building RAG pipelines, AI agents, and data extraction workflows. ## Overview Crawl4AI turns the web into structured Markdown suitable for downstream AI processing. It supports both programmatic Python use and a command-line interface, with optional LLM-driven structured data extraction and CSS/XPath-based schema extraction. ## Key Capabilities ### Markdown Generation - Produces clean, structured Markdown with headings, tables, and code blocks - Fit Markdown mode applies heuristic filtering to remove noise and irrelevant content - Citation-aware output that converts links into numbered references - BM25-based content filtering for query-relevant extraction - Pluggable markdown generation strategies ### Structured Data Extraction - LLM-driven extraction via LiteLLM, supporting both open-source and proprietary models - JSON schema-based extraction using CSS selectors and XPath - Chunking strategies (topic-based, regex, sentence-level) for targeted processing - Cosine similarity search for semantic content matching ### Browser Integration - Async browser pool powered by Playwright - Managed browser mode using user-owned browser profiles with saved auth states and cookies - Remote browser control via Chrome DevTools Protocol - Proxy support with authentication - Stealth mode to reduce bot detection - Multi-browser support (Chromium, Firefox, WebKit) - Dynamic viewport adjustment ### Crawling Features - JavaScript execution with async/sync wait support for dynamic content - Screenshot capture during crawls - Media extraction (images, audio, video, responsive srcset/picture formats) - Raw HTML and local file processing - Comprehensive link extraction including iframe content - Lazy load handling and full-page scanning for infinite scroll - Hook-based customization at every crawl step - Built-in caching ### Deployment - Dockerized FastAPI server for API-based workflows - JWT token authentication (enabled by default in recent versions) - Monitoring dashboard and interactive playground at `/dashboard` and `/playground` - MCP integration for AI tool connections - Multi-architecture Docker images (AMD64/ARM64) - Cloud deployment ready ## Installation ```bash pip install -U crawl4ai crawl4ai-setup ``` For manual Playwright installation: ```bash python -m playwright install --with-deps chromium ``` ## Usage ### Python API ```python import asyncio from crawl4ai import AsyncWebCrawler async def main(): async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com") print(result.markdown) asyncio.run(main()) ``` ### CLI ```bash # Basic crawl crwl https://example.com -o markdown # Deep crawl crwl https://example.com --deep-crawl bfs --max-pages 10 # LLM extraction crwl https://example.com -q "Extract all product prices" ``` ### Docker ```bash docker pull unclecode/crawl4ai:latest docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest ``` Then visit `http://localhost:11235/dashboard` for the monitoring UI or `http://localhost:11235/playground` for the interactive test environment. ## Project Details - Repository: unclecode/crawl4ai - PyPI: crawl4ai - Documentation: https://docs.crawl4ai.com/ - Discord community: https://discord.gg/jP8KfhDhyN - Sponsors: https://github.com/sponsors/unclecode The project emphasizes open availability and affordability, positioning itself as a self-hosted alternative to commercial web extraction APIs. It has accumulated over 50,000 GitHub stars and actively maintains security through regular releases.