bncc-benchmark

# bncc-benchmark Public LLM hallucination benchmark regarding the BNCC (Base Nacional Comum Curricular). It measures, with open methodology and published raw data, how much language models invent BNCC codes and texts when responding without access to the structured source, and how much the problem disappears with grounding via bncc.dev (MCP and API). ## Official Round A `oficial-seca-2026-08` round measured **19 models × 900 responses each** (17,100 raw responses). The exact text of a BNCC skill is asked without providing access to the source; the rate of responses faithful to the official text varies from 90% to 0% depending on the model. ### Top (5 models) | Model | Score | Faithful Text | Accepted False Code | |---|---|---|---| | GPT-5.6 Sol · OpenAI | 86.4 | 90% | 25% | | Claude Fable 5 · Anthropic | 80.2 | 77% | 3% | | Gemini 3.1 Pro · Google | 76.0 | 61% | 4% | | Claude Opus 5 · Anthropic | 75.0 | 67% | 4% | | GPT-5.6 Luna · OpenAI | 73.8 | 75% | 43% | *Score* is the average of five dimensions: recognizing real codes, refusing false ones, text fidelity, inverse lookup, and correct citation in open generation. **Two lessons from the ranking:** (1) a high score does not imply confidence — the first place still accepts 25% of false codes as real; (2) getting the text right and refusing inventions are distinct skills, which is why points are spread out. GPT-5.6 Luna and Claude Fable 5 have similar fidelity (75% vs 77%), but the former accepts 43% of invented codes while the latter accepts only 3%. ## Interventions and held-out The intervention study (8 models, 300 items, three paired conditions) showed: - Without source: **31.9%** of responses with hallucination - Data in prompt (without tool): **0.2%** - Consulting the MCP (bncc.dev): **2.3%** There is a **private held-out set** (never published) to detect if superior performance in the public set stems from memorization or genuine learning of the BNCC. ## What is measured | Task | Typical Question | Measures | |---|---|---| | A · direct lookup | "What is the text of skill EF67LP08?" | invented or swapped text | | B · existence | "Does skill X exist in the BNCC?" | acceptance of plausible-false codes | | C · open generation | "List 5 Mathematics skills for 7th grade" | invented codes in real use | | D · inverse lookup | "What is the code for this skill?" (given text) | memorization in the reverse direction | The answer key is the verified dataset from bncc.dev (`@bncc/dados`, 1,721 learnings traceable to the official document). The false codes for task B are constructed from legitimate gaps in the official numbering. ## Repository Structure ``` harness/ benchmark code (generator, runner, evaluation, aggregation) items/ versioned item bank results/ raw responses (JSONL) and aggregates, per round METHODOLOGY.md complete protocol DECISIONS.md numbered design decisions ``` ## Usage ```bash pnpm install pnpm test pnpm generate pnpm execute --round smoke --models claude-haiku --limit 10 pnpm evaluate --round smoke pnpm aggregate --round smoke pnpm aggregate --round smoke --verify # check used by CI ``` Local execution; CI only runs typecheck, tests, and result consistency verification. Provider keys in `.env` (never committed). ## Licenses and maintenance Code: MIT. Items, results, and methodology: CC BY 4.0. Maintained by [Profy](https://www.profy.ai/) — there is a declared conflict (Profy operates products that use LLMs on the BNCC), compensated by total transparency: methodology, items, raw responses, and judgments are public and recalculable, and the CI rejects any manually edited score. Published rounds are immutable.