Development record

Ingesting and consulting sources are abcd verbs, and no corpus entry is a keyword stub

intentdraftsminoritd-2609292108089653

Typed links: related_issues iss-27 (sources tooling into the Go core; resolved by itd-76 for the corpus contract, and the ruling plans what remains) and iss-55 (the placeholder entries to backfill); builds_on itd-76 (the abcd source verb family this extends).

Press Release

Ingesting a source and consulting the sources corpus are abcd verbs, and every entry the corpus holds carries the text it claims to hold. The deterministic half of ingest and consult moves out of the command pages into the binary: a search over the corpus that answers by key, a text-quality check that refuses a keyword stub whose entry claims a fetchable source, and the backfill of the placeholder entries registered before the text was stored. "I asked the corpus a question and got back six entries that were only keywords," said Maya, a researcher-developer. "Now a stub is refused at ingest, the old ones carry their text, and the search is the same every time I run it."

Why This Matters

itd-76 moved the corpus contract into the binary as the abcd source verb family (init, add, ledger, declassify, sync-banlist, cite-check), and iss-27 was resolved on that delivery. Ingest and consult themselves are still command pages that an agent follows by hand: the page says "abcd fetches and converts nothing", consulting is "plain search", and the text-quality check is a step in prose. Six public entries (iss-55) are keyword stubs with no document text, which yielded nothing when the corpus was searched, so the corpus claims coverage it cannot deliver. The product thinker ruled on 2026-09-29 to plan the sources tooling into abcd's core with ingest and consult as abcd verbs (ruling J5), with the backfill of the placeholder entries as part of the same intent (ruling J6).

What's In Scope

A consult verb: a read-only search over the corpus that answers by key and class, under the confidentiality hard rule (a confidential entry is named by key only).
The ingest quality check as a verb: a stored text that is only keywords, for an entry that claims a fetchable source, is refused at source add and reported by a corpus lint.
The backfill of the six stub entries iss-55 names, and the on-disk PDF attached to the research-integrity paper entry, through the ingest path.

What's Out of Scope

A pluggable retrieval back end (iss-26, ruled "decide later" the same day).
Fetching and converting inside the binary, unless the planning interview decides it (see Open Questions).

Mechanism

Prompted (the claim-recording gradient): why the authors expect this to work, as a falsifiable "we expect X because Y" — not the outcome restated. Replace this line with the claim, or with the exact token None stated. alone on its line to record the claim as considered and declined.

Scope Conditions

Required (the claim-recording gradient): the population, platform, scale, or assumptions this claim holds under, one per top-level bullet — abcd intent plan stamps each with a persistent identity. Replace this line with those bullets, or with the exact token None stated. alone on its line.

Acceptance Criteria

Seeded by the run from the two records and the ruling, unconfirmed: the planning interview walks each one.

Given a corpus, when the consult verb runs with a query, then it lists the matching entries by key and class, names a confidential entry by key only, and writes nothing.
Given a text that is only keywords, for an entry that claims a fetchable URL, when source add is given it, then the add is refused, naming the entry.
Given the corpus after the backfill, when the corpus lint runs, then it reports no stub entry, and a consult over the six entries iss-55 names returns document text rather than keywords.

SOTA

Surveyed on 2026-10-23; sources, evidence tiers and the local fit risks are in the sources-provenance SOTA note.

Extraction quality is measured. On trafilatura's own 2026-10-02 benchmark (990 documents) trafilatura scores F 0.926, readability-lxml 0.853, jusText 0.862, newspaper4k 0.803 (trafilatura evaluation page). A Go port, go-trafilatura v2.2.6 (Apache-2.0, Go 1.26 or newer), exists and would be a Go dependency needing sign-off.
The keyword-stub test needs no model. jusText classifies a block by stopword density (bad below 0.30, good above 0.32 for long blocks), link density (above 0.2 is boilerplate) and length (70 and 200); a text that is only keywords has near-zero stopword density (jusText algorithm page). The density is computed against a per-language stoplist (the tool is called with get_stoplist("English")) and the bounds are block-level defaults for HTML, so the check must pick the stoplist by the entry's language, or skip with a warning when the language is unknown, and recalibrate the bounds for whole stored texts.
PDF text in pure Go is weak. pdfcpu extracts images, fonts and metadata, not plain text; ledongthuc/pdf extracts text but calls itself incomplete; poppler's pdftotext converts to plain text in reading-order or -layout mode (the three projects' own pages).
Consult stays grep-first (2026-07-08 ruling); the named upgrade, SQLite FTS5, is reachable CGO-free through modernc.org/sqlite (BSD-3-Clause, SQLite 3.54.1) and would be a dependency needing sign-off.
adr-38 bars an implicit fetch, not an explicit verb, so an ingest that fetches on command is within the rule; the page's "abcd fetches and converts nothing" predates it.
Path 2: consult and the stopword-density quality check (at source add and in source lint) are native; the backfill of the six stub entries is a documented act against a person's corpus, verified by source lint rather than by a build criterion; ingest <url|file> is explicit and converts through installed tools on PATH (pdftotext, pandoc, the trafilatura CLI), and with none present stages the bytes and tells the agent to convert and re-run with --text. The seam is the converter interface (bytes and media type in, text out), so go-trafilatura can replace the HTML rung under its own sign-off. The independent fit-challenge runs at plan time.

Open Questions

(SOTA 2026-10-23) The stopword threshold and token floor want calibrating on the real corpus before they harden; does the first cut ship as a lint warning and become a refusal once the six stubs and a sample of good entries have been scored?
Does ingest fetch and convert in the binary, or does the agent keep the fetch and the conversion while the binary does everything deterministic? The ingest page states "abcd fetches and converts nothing" today, and adr-38 bars an implicit fetch.
The consult verb's name: a sub-verb of abcd source, or its own top-level verb beside the /abcd:consult page (since retired into /abcd:library).
The backfill's home: the six entries live in a person's own corpus, not in this repository, so the backfill may be a documented act rather than a criterion the build can meet.

Audit Notes

Empty. Populated by intent-auditor when intent moves to shipped/.

Frontmatter

id	itd-2609292108089653
type	intent
lifecycle	drafts
severity	minor

.abcd/development/intents/drafts/itd-2609292108089653-ingesting-a-source-and-consulting-the.md
open on GitHub ↗ · commit history ↗

References1

itd-76abcd Lets You Consult Sources You Cannot Cite — and Remembers Every Debtbuilds on

Referenced by10

iss-27corpus tooling graduates into the Go core: the sources-corpus contract (per-source folders with location-as-classification, CSL-JSON metadata, append-only provenance ledger, banlist projection) is proven by the user-tier script MVP; absorb it as a native abcd sources/ingest verb family, converging with the itd-36 provenance substrate (internal/core/provenance: licence detection, citation generation, source-hash registry) rather than porting standalone. The scripts remain the reference implementation until the native verbs are wired; /abcd:consult and /abcd:ingest keep one surface across the swap.related
iss-55Six public corpus entries are keyword stubs with no document text — five practitioner SDLC pieces and one harness-engineering post — and they yielded zero extraction candidates in the 2026-07-09 tool-extraction pass, so the corpus claims coverage it cannot deliver. The fix serves retrieval honesty: backfill the full texts via the ingest path, and attach the on-disk PDF to the already-registered research-integrity paper entry so its text is queryable too. Detector: a corpus lint flags any text.md stub whose ledger entry claims a fetchable URL; acceptance is a lint run with zero such stubs and a consult query over the six entries returning document text rather than keywords.related
iss-2610262031355278itd-2609292108089653 refuses stubs at source add, while the brief's add for the sources corpus is a library sub-verbrelated
iss-2610262034111313draft itd-2609292108089653 calls registering a document into the sources corpus 'ingesting a source'; the library surface's ingest distils into the memory store and the corpus's add registersrelated
iss-2610262039000162itd-2609292108089653 refuses a stub at source add; the surfaces register folds the source command into libraryrelated
iss-2610262040412521itd-2609292108089653 registers sources with `source add`; the library glossary registers them with /abcd:library add.related
iss-2610262042462569itd-2609292108089653 refuses stubs at source add; itd-2610091918433290 says the sources command was merged into libraryrelated
iss-2610262042473980Ingest means importing shared bibliography entries in itd-126 and storing a source's text in itd-2609292108089653related
iss-2610262043030522The sources-corpus draft names its ingest, consult and lint 'source add', a consult verb and a corpus lint; itd-39 says the store is already reached as library ingest, ask and lint.related
iss-2610262043323769itd-2609292108089653 builds ingest and consult as their own verbs; itd-2610090831227812 merged consult, ingest and source into libraryrelated

mentions5

adr-38Implicit checks are disk-only — the network answers only an explicit ask
dcn-26092900000000112026-09-29 — The product thinker answered the twenty-six Group J rulings (plan or close), J1 to J26 (recorded by lane recGroupJ of autonomous run A, for orchestrator abcd-fc). The answers are kept verbatim in the local-tier file `.abcd/.work.local/scratch/reports/rulings-answered-2609-29.md`, section "Group J", dated 2026-09-29, and are restated here because that file is not committed. PLAN: the installer script becomes a minimal starter with an ownership-checked hand-over, planned as a new intent with its planning interview owed (J2, iss-377, draft itd-2609292106557115); token metering and size classes are planned as one intent (J3, iss-2608301744251874 and iss-2608301856299268), and session token accounting in the history store joins that metering intent (J4, iss-2608220150157508; all three in draft itd-2609292107351737); the sources tooling moves into abcd's core with ingest and consult as abcd verbs (J5, iss-27, whose corpus half itd-76 already delivered), and the backfill of the placeholder source entries is part of that intent (J6, iss-55; both in draft itd-2609292108089653); the grill hands off to an installed external interview skill and falls back to its own (J11, iss-165, draft itd-2609292108373494); an opt-in local-model prompt sanitiser is planned (J14, iss-2608261543489261, draft itd-2609292109005937); decision-to-transcript links are planned, with the anchor kind and the link's home settled in the planning interview owed for draft itd-171 (J16, iss-2608290819228175); the dredge synthesis, its write-ups as their own memory source class, is planned together with draft itd-25, whose planning interview is owed (J19, iss-2609211905340006); one provenance register across ingest and vendoring is planned together with draft itd-26, whose planning interview is owed (J20, iss-2609211905346507); release retention, keeping the newest release per version line, is automated by planning draft itd-70 (J21, iss-282); a rules-backend seam with an opt-in CARL adapter is planned, the native loader staying the default (J22, iss-64, draft itd-2609292109214516); core-owned managed pre-commit gates are planned with draft itd-62 (J23, iss-84); a behavioural end-to-end scenario suite per verb family is planned (J24, iss-48, draft itd-2609292109475690). Every draft stays in drafts/ until a person plans it, and each source issue links its draft and names that planning interview in its deferral. BUILD: the teaching plane, the rules-loader safety domain generated from the hazard registry, is built as a lane, the registry being the single source (J10, iss-151 and itd-103; a code lane owns it). RULED DETAILS: `source_kind` carries both, as two separate labels, one for the tool and one for the route (J13, iss-2608230752354928); the prompt router's removal signal is the full list of active domains every time, a domain's absence meaning it has stopped (J15, iss-2608261550580260); setup offers an outside AI service (for example OpenRouter) at install, skippable, once the API adapter ships (J18, iss-2609081951416843, which waits on the API adapter); the salvage-hook timeout is 120 seconds, the wait must show a counter or a message while it runs and never stall silently, and every per-event timeout is pinned in a test (J25, iss-323). J13, J15 and J25 are carried by code lanes. CLOSE: the quick-tunnel preview protocol is closed unless the need recurs (J26, iss-2608230617385431, wontfix with that reason). PARKED ON A TRIGGER: agent payment protocols are kept under watch, not now, revisited as the protocols mature (J8, iss-138); the findings-only skill waits for the skill format to be versioned and stewarded (J9, iss-139); the disposition worksheet as an intent and a site page waits until after the first study (J17, iss-2609021815529020). DECIDE LATER: the Homebrew tap stays parked (J1, iss-380); pluggable search back ends (J7, iss-26); forge-backed record numbering (J12, iss-2608210737264758).
dcn-26092900000000202026-09-29 — The product thinker answered the twenty-six Group J rulings (plan or close), J1 to J26 (recorded by lane recGroupJ of autonomous run A, for orchestrator abcd-fc). The answers are kept verbatim in the local-tier file `.abcd/.work.local/scratch/reports/rulings-answered-2609-29.md`, section "Group J", dated 2026-09-29, and are restated here because that file is not committed. PLAN: the installer script becomes a minimal starter with an ownership-checked hand-over, planned as a new intent with its planning interview owed (J2, iss-377, draft itd-2609292106557115); token metering and size classes are planned as one intent (J3, iss-2608301744251874 and iss-2608301856299268), and session token accounting in the history store joins that metering intent (J4, iss-2608220150157508; all three in draft itd-2609292107351737); the sources tooling moves into abcd's core with ingest and consult as abcd verbs (J5, iss-27, whose corpus half itd-76 already delivered), and the backfill of the placeholder source entries is part of that intent (J6, iss-55; both in draft itd-2609292108089653); the grill hands off to an installed external interview skill and falls back to its own (J11, iss-165, draft itd-2609292108373494); an opt-in local-model prompt sanitiser is planned (J14, iss-2608261543489261, draft itd-2609292109005937); decision-to-transcript links are planned, with the anchor kind and the link's home settled in the planning interview owed for draft itd-171 (J16, iss-2608290819228175); the dredge synthesis, its write-ups as their own memory source class, is planned together with draft itd-25, whose planning interview is owed (J19, iss-2609211905340006); one provenance register across ingest and vendoring is planned together with draft itd-26, whose planning interview is owed (J20, iss-2609211905346507); release retention, keeping the newest release per version line, is automated by planning draft itd-70 (J21, iss-282); a rules-backend seam with an opt-in CARL adapter is planned, the native loader staying the default (J22, iss-64, draft itd-2609292109214516); core-owned managed pre-commit gates are planned with draft itd-62 (J23, iss-84); a behavioural end-to-end scenario suite per verb family is planned (J24, iss-48, draft itd-2609292109475690). Every draft stays in drafts/ until a person plans it, and each source issue links its draft and names that planning interview in its deferral. BUILD: the teaching plane, the rules-loader safety domain generated from the hazard registry, is built as a lane, the registry being the single source (J10, iss-151 and itd-103; a code lane owns it). RULED DETAILS: `source_kind` carries both, as two separate labels, one for the tool and one for the route (J13, iss-2608230752354928); the prompt router's removal signal is the full list of active domains every time, a domain's absence meaning it has stopped (J15, iss-2608261550580260); setup offers an outside AI service (for example OpenRouter) at install, skippable, once the API adapter ships (J18, iss-2609081951416843, which waits on the API adapter); the salvage-hook timeout is 120 seconds, the wait must show a counter or a message while it runs and never stall silently, and every per-event timeout is pinned in a test (J25, iss-323). J13, J15 and J25 are carried by code lanes. CLOSE: the quick-tunnel preview protocol is closed unless the need recurs (J26, iss-2608230617385431, wontfix with that reason). PARKED ON A TRIGGER: agent payment protocols are kept under watch, not now, revisited as the protocols mature (J8, iss-138); the findings-only skill waits for the skill format to be versioned and stewarded (J9, iss-139); the disposition worksheet as an intent and a site page waits until after the first study (J17, iss-2609021815529020). DECIDE LATER: the Homebrew tap stays parked (J1, iss-380); pluggable search back ends (J7, iss-26); forge-backed record numbering (J12, iss-2608210737264758).
iss-26sources corpus retrieval seam: the MVP is the current native store (per-source folders, text.md extraction, grep-based consult — shipped) and retrieval/processing becomes a pluggable seam with opt-in deeper backends, mirroring the spec seam pattern (native minimal default, adapter for depth): a RAG backend (RAG-Anything, already downloaded locally, has its own content processor that could replace pandoc/pdftotext extraction and serve semantic retrieval) and/or a CMS backend. Adapter contract, not a rewrite: consult and ingest skills keep one surface; the backend is configuration.
spc-2610100613109045a-person-who-types-abcd-sees-only-the-commands-people-use