defuddle

Defuddle is a JavaScript library that extracts the main content from web pages, cleaning up clutter like comments, sidebars, headers, and footers. It returns cleaned HTML or Markdown, making it useful for web clipping, reading modes, or content analysis. ## Key Features - **Content Extraction**: Automatically detects and extracts the primary content from a page, removing non-essential elements. - **HTML Standardization**: Normalizes headings, code blocks, footnotes, math, and callouts for consistent output. - **Metadata Extraction**: Pulls title, author, description, publication date, and schema.org data. - **Multiple Bundles**: Core (browser), Full (adds math/Markdown conversion), and Node.js (works with any DOM implementation). - **CLI Tool**: Parse URLs, files, or stdin directly from the terminal. - **Debug Mode**: Detailed logging and removal tracking for diagnosing extraction issues. - **Configurable Pipeline**: Toggle individual processing steps (e.g., hidden element removal, image removal). ## Usage Examples ### Browser ```javascript import Defuddle from 'defuddle'; const defuddle = new Defuddle(document); const result = defuddle.parse(); console.log(result.content); // Cleaned HTML ``` ### Node.js (with linkedom or JSDOM) ```javascript import { parseHTML } from 'linkedom'; import { Defuddle } from 'defuddle/node'; const { document } = parseHTML(html); const result = await Defuddle(document, 'https://example.com/article', { markdown: true }); ``` ### CLI ```bash npx defuddle parse page.html --markdown npx defuddle parse https://example.com/article --json cat page.html | npx defuddle parse --output result.html ``` ## Response Object Returns an object with properties like `title`, `author`, `content`, `description`, `domain`, `favicon`, `image`, `language`, `published`, `site`, `wordCount`, and more. ## Options Key options include `markdown` (convert to Markdown), `debug` (enable logging), `removeHiddenElements`, `removeSmallImages`, `contentSelector` (bypass auto-detection), and `useAsync` (allow third-party fallback for SPAs). ## Development Build with `npm install && npm run build`. The project uses a pipeline of steps for content scoring, element removal, and standardization. ## Third-Party Services When `useAsync` is enabled and no local content is found (e.g., client-side rendered SPAs), Defuddle may fetch from third-party APIs like FxTwitter for X/Twitter content. Disable with `useAsync: false`. ## Debugging Enable `debug: true` to get a `debug` field with `contentSelector` (CSS path of main content) and `removals` (list of removed elements with reasons). This helps diagnose why content is being incorrectly removed or kept.