eurlex-builder

eurlex-builder is a configurable Python command-line pipeline for building research-ready datasets from EUR-Lex / Cellar legislative material. It retrieves directives, regulations, decisions, communications, and other EU acts, then extracts text at selectable granularity: whole articles, numbered paragraphs, or lettered points. The tool is aimed at quantitative legal research, NLP, classification, and network analysis of EU legislation. The pipeline produces a DuckDB working database plus four Parquet tables: works, text_units, relations, and eurovoc. It supports two main query modes: fixed mode, where users provide CELEX IDs or procedure numbers, and descriptive mode, where users select document types, date ranges, and optional EuroVoc keyword filters. It handles multiple EUR-Lex HTML structures and includes PDF fallback extraction for older or non-machine-readable documents using Docling and pymupdf. Key capabilities include sub-article decomposition, boilerplate stripping, recital/article/annex extraction, inter-document relation capture such as citations, amendments, legal bases, repeals, and consolidations, and optional translation of non-English content using Helsinki-NLP Opus-MT revisions. Translation is guarded with token-bounded retries and quality checks, and rejected outputs remain in the source language rather than replacing the original text. The tool emphasizes reproducibility and resumability. A single YAML file defines the dataset, while DuckDB stores checkpoints, run manifests, validated configuration hashes, dependency versions, and completion status. CLI commands include run, translate, enrich, status, and validate. Enrichment can add post-hoc metadata via SPARQL, including entry-into-force dates, ELI, author institutions, subject matter, procedure details, EuroVoc descriptors, and repeal relations. The validate command performs read-only integrity checks without modifying the database. The README notes that full-text retrieval works broadly, but granular extraction is most reliable for regulations, directives, and decisions. Communications extract well but have known gaps, while proposals, staff working documents, and case law may lose structural units or produce no text units. This makes the project especially useful for researchers building structured, auditable corpora of EU legislative texts.