Run it yourself

Optional. The exercises need only a browser; this page is for rerunning or extending the example.

What is in this repository

outputs/structure/   the extraction prompt, Trimen's structured record, three audit examples, the structuring runbook
outputs/identity/    the merge rules and loop, the approach document, a sample of twelve clusters, the first 25 ledger entries
outputs/ground/      place-grounding cache rows for Victoria, South Africa, Penang, Perth; Brewster's record
outputs/ground-test/ twenty entries for the grounding test, the answer key with MCP lookups, the Gemini run and its scoring
outputs/audit/       the structuring error catalogue, the every-tenth reviewed pairs, the September 2026 project review, the running fixes list
outputs/atlas/       kg_stats for both corpora, the atlas meta.json, the Wodehouse bridge records, the essay on grounding
annotate/            the mark-up exercise (static HTML + bios.json)
review/              the decision exercise (static HTML + cases.json)
source-pack/         Trimen's seven Colonial Office List entries
archive/tropical/    the earlier draft built on The Tropical Agriculturist, kept for reference

The dataset

Everything here is drawn from Imperial Careers (data CC0, code MIT, DOI 10.5281/zenodo.21079938). The finished graph is committed there as JSONL under data/kg/graph_stage3/ and data/iol/graph_stage3/; the README shows how to load it into LadybugDB or Neo4j in two commands. The pipeline code, the grounding caches and the method documents are in the same repository.

Rebuilding the exercise

python3 scripts/build_review_cases.py      # cases.json for the decision exercise
python3 scripts/demo_ditto_fix.py          # before/after records and bounded checks
python3 scripts/extract_rules_examples.py --col-matching ~/col_matching   # real rule-parser output and coverage

No network needed.

Running the grounding test

outputs/ground-test/bios.md has the twenty entries. For the chatbot column, paste them into any model without tools. For the retrieval column, use the WikidataMCP server (Streamable HTTP endpoint https://wd-mcp.wmcloud.org/mcp) from a coding agent, or its public search endpoint. Two things to know:

  • Use the vector search, not the string search. Wikidata’s ordinary search API matches labels by string similarity. The MCP server’s search_items uses semantic embeddings. It is rate limited; cache what you retrieve.
  • No identifiers from memory. Tell the agent, and check its output. Resolve every identifier you are given before you keep it; wbgetentities with a list of ids is enough to see what they are.

Running the structuring prompt

outputs/structure/extraction-prompt.txt is the system prompt. Send it with one entry as the user message to any OpenAI-compatible endpoint and ask for JSON. The Imperial Careers repository’s col_match/kg/extract.py is the production version, with the validator that drops any place or year not present in the source.

The endpoint can be a hosted service such as OpenRouter (an API key, pay per token), a model you serve yourself with vLLM on a desktop GPU, or a batch job on a research cluster. The glossary compares the three as the project used them, with a worked cost estimate.

Credentials

No file in this repository contains a credential, and none is needed for the exercises. If you configure an agent with an API key, keep it out of the repository.