web-core

web-core is a shared web infrastructure package for Python, described as powering the wet-mcp MCP server and other downstream applications. It bundles several web-facing capabilities behind a typed public API, and the README notes that site-specific selectors are deliberately left to consumer applications — the package itself provides generic infrastructure only, with per-domain cookies and selectors supplied through environment variables. Capabilities listed in the README: - SearXNG search: a cross-process singleton runner (file-lock based, ensuring one instance across all Python processes) plus a client that performs retries, result deduplication and domain filtering. - Multi-strategy scraping: a LangGraph state-machine agent that escalates across strategies. Strategy modules shipped include API-direct fetching, basic HTTP, TLS fingerprint spoofing via curl-cffi, headless rendering via Crawl4AI, Patchright stealth-browser rendering, remote rendering through a RenderClient, and a CapSolver-backed CAPTCHA strategy. Strategies are attempted in cache-recommended order, responses are validated, and failures escalate automatically — including past under-rendered JavaScript shells to a render backend. A StrategyCache tracks per-domain performance. - SSRF-safe HTTP client: a DNS-pinned httpx client, URL normalization, tracking-parameter stripping and domain validation helpers. All outbound HTTP is intended to go through safe_httpx_client to guard against DNS rebinding. - Browsers and remote rendering: a BrowserProvider protocol plus a Patchright (undetected Playwright) provider, a BrowserlessClient for self-hosted /content endpoints, and a Cloudflare Browser Rendering client, aimed at slim containers that offload JavaScript rendering. - robots.txt compliance: a per-domain cached check before fetching. - LLM selector inference (optional): when built-in selectors fail, a provider can be auto-detected from whichever API key is present (Google Gemini, OpenAI, Anthropic, xAI); if no key is set, inference is skipped silently. A custom llm_caller can also be injected. Gemini can be routed through Vertex AI via project/location variables. - External API adapters: typed, SSRF-safe clients for Google Drive folder/file fetching and MangaDex (manga, chapters, images). Installation is from PyPI as n24q02m-web-core (uv add), with a v2.x stable line. Configuration is entirely environment-variable driven and every variable is optional — omitting one disables the feature it controls, and no variable is required to import or use the package. Documented variables cover SearXNG URL and run user, per-domain cookies (JSON) for scraping, and the LLM provider keys plus model override for selector inference. Remote render credentials are passed as constructor arguments instead. Architecture is organized under src/web_core with layered modules: http (SSRF-safe primitives, URL utilities), search (client, models, runner), scraper (agent, base strategy, cache, robots, selector inference, state, utils, and the strategies subpackage), browsers (protocol, patchright, browserless, cf_rendering) and adapters (google_drive, mangadex). The package ships a py.typed marker for PEP 561 typing. Design decisions highlighted include DNS pinning for SSRF protection, automatic strategy escalation, the cross-process SearXNG singleton, and use of Protocol-based structural typing so browser and render implementations need not inherit from a base class. Development targets Python 3.13 with uv, optional mise task shortcuts, pre-commit hooks, ruff for lint/format, ty for type checking and pytest with coverage — the test suite enforces a 95% coverage threshold and uses asyncio_mode auto. Test files mirror the source module structure. The project is licensed under Apache-2.0.