katana

Katana is a Go-based web crawling and spidering framework intended to be used both directly and as part of automated workflows. It accepts targets as a single URL, comma-separated URLs, a list file, or piped STDIN input, and can emit discovered URLs and metadata to STDOUT, a file, or JSONL. Installation options include go install, prebuilt release binaries, and a Docker image. The standard crawl mode uses Go's HTTP library and does not render JavaScript or the DOM. The optional headless mode uses Chrome to render pages, execute JavaScript, and discover browser-loaded or asynchronous endpoints. Headless configuration includes system Chrome selection, a custom Chrome path or debugger WebSocket URL, browser data directories, incognito and sandbox controls, visible-browser mode, additional Chrome options, XHR extraction, and page-load strategies such as heuristic, load, DOMContentLoaded, networkidle, and none. A DOM wait time can be configured for JavaScript rendering. Katana can parse and crawl endpoints from JavaScript files and optionally use jsluice parsing. It supports experimental automatic form filling, form and field extraction, custom form configuration, and automatic captcha detection with external solving through CapSolver. The documented supported captcha types are reCAPTCHA v2, reCAPTCHA v3, reCAPTCHA Enterprise, Cloudflare Turnstile, and hCaptcha. Crawl behavior can be constrained by depth, duration, maximum pages per domain, response size, timeouts, retries, redirect handling, proxies, custom headers or cookies, custom resolvers, and depth-first or breadth-first strategies. It can also process robots.txt and sitemap files, detect technologies, and optionally randomize the TLS client hello. Scope controls include the predefined rdn, fqdn, and dn fields, regular-expression include and exclude rules, host-scope disabling, external-URL display, and exclusion of CDNs, private IP ranges, CIDRs, IPs, or regex-matched hosts. Output filtering supports URL regexes, extension matching or filtering, DSL-based response conditions, page-type filters, query-parameter normalization, similar-URL filtering, and duplicate-content filtering. Page-content similarity can use simhash, TF-IDF, or BM25 with configurable thresholds and per-cluster processing budgets. Rate and execution controls cover concurrent fetchers, parallel inputs, request delays, and global or per-host rate limits per second and per minute. Output can use configurable fields or templates, store HTTP requests and responses, omit raw data or bodies, and run in silent, verbose, or debug mode. Crawls can be resumed from a resume configuration file. An optional knowledge-base feature provides ML-based page-type and form classification using an automatically downloaded model. It can also extract secrets, optionally validate them with live provider calls, and classify REST, GraphQL, SOAP, and XHR endpoints. The project documentation reminds users to use the tool responsibly and that they are responsible for their own usage.