llm-wiki-compiler

llmwiki (npm package llm-wiki-compiler) is a command-line knowledge compiler that turns raw source material into an interlinked markdown wiki with citations, metadata, and review state. It is a concrete implementation of Andrej Karpathy's LLM Wiki pattern: instead of retrieving raw chunks at query time, it compiles knowledge once into durable pages that accumulate structure and provenance. What it does A two-phase LLM pipeline extracts concepts from sources and then generates typed pages such as concept, entity, comparison, and overview. Paragraphs and claims cite source files and line ranges, and a lint command validates those links. The default profile preserves a concepts-and-queries layout; optional profiles add domain-specific types and workflows without branching the compiler itself. Key capabilities - Configurable Lifecycle Profiles: a validated .llmwiki/profile.json declares typed entities, fields, directed relations, lifecycle states, transition evidence, trust gates, multi-stage workflows, hash-pinned artifacts, connectors, content tiers, and retrieval policy. Rules are enforced by the runtime rather than left as prompt conventions, and invalid profiles fail closed. - Installable domain templates: built-in templates such as autosci (papers, ideas, experiments, manuscripts, evidence artifacts, Crossref import) and newsroom (articles, desks, bylines, editorial workflows). Templates contain configuration and examples, not executable plugin code. Signed template distributions use Ed25519 signing, key rotation, and revocation, with explicitly trusted taps for discovery. - Hybrid retrieval: semantic chunk search, BM25 reranking, and wikilink graph expansion build compact evidence packs for queries and agents. - Local viewer: a read-only browser UI with search, page metadata, graph exploration, source-freshness badges, and citation chips. Version 1.3 adds four themes. - Review policy: generated pages can be held as candidates when confidence, contradiction, schema, or provenance rules trip. - Freshness repair: lint and next surface stale or orphaned pages; refresh --stale repairs changed knowledge without compiling unrelated new sources. - Eval harness: reports health score, per-page health distribution, wikilink-graph health, citation coverage and precision, corpus stats, and regression deltas, with an optional judge model for citation support. - MCP server: exposes ingest, compile, query, lint, read, status, eval, context-pack, and OKF exchange tools to MCP-compatible agents. - SDK: createWiki({ root }) drives ingest, compile, query, context, status, export, eval, and OKF import/export from TypeScript. - Open Knowledge Format exchange: export and import OKF bundles; external imports are staged through the review queue by default, with trusted bundles writable live explicitly. - Other exports: JSON, JSON-LD, GraphML, Marp slides, and llms.txt. - Provider portability: Anthropic, Claude Agent SDK local login, OpenAI Codex CLI local login, OpenAI-compatible servers, Ollama, GitHub Copilot, Atlas Cloud, OrcaRouter, and local OpenAI-compatible runtimes. Project layout A project keeps raw inputs in sources/, compiled markdown in wiki/ (concepts, queries, profile-declared typed pages, graph stores, outputs, generated index), and compiler state under .llmwiki/ (profile, config, schema, state, candidates, workflows, eval history). Artifacts are hash-pinned files and manifests; log.md is an activity journal. Quality and safety model The project emphasizes auditable generated knowledge: review-before-write policies, runtime-enforced profile floors, untrusted external connector data staged as fenced review candidates, path-confined and hash-verified artifacts, fail-closed configuration, source confinement, explicit freshness states, and CI quality gates via lint and eval. Scale notes Incremental compilation skips unchanged sources, parallel compile runs extraction and generation under a configurable concurrency cap, chunk-level embeddings narrow large wikis before reranking, content-hash-aware embedding updates avoid recomputation, batch embedding reduces latency, binary embedding storage handles stores above the 64 MiB JSON limit, and lexical fallback keeps query and context workflows usable when a provider lacks an embedding endpoint. Requirements and usage Node.js 24 or newer is required. Typical usage is npm install -g llm-wiki-compiler, set a provider API key, then run quickstart on a file or URL, query the compiled wiki, and open the viewer. The README notes the tool suits durable project or domain wikis (research folders, codebase docs, team handbooks, standards, decision logs) and is a less ideal fit for high-churn raw logs where plain search suffices. It is described as early software.