Cleaning, Structuring and Grounding Historical Data

One dataset, start to finish, with a coding agent

Jim Clifford, University of Saskatchewan

15 October 2026

Thousands of pages like this…

…became the moving atlas on the next slide.
The Colonial Office List for 1913 (52nd publication), pp. 561–562. Scan digitised by Google, from the Internet Archive.

0–1. One simple point before the video: this pipeline turned thousands of printed pages like these into the animated atlas that follows. Two facing pages of the Record of Services hold about forty compressed biographies; the project OCR’d 85 volumes, 1867–1966 (Colonial Office Lists, with the 1935 Dominions Office List and the 1951–65 Commonwealth Relations Office Lists), plus the India Office List. The List was first published in 1862; the project’s run starts in 1867. Then play the video.

Open the video directly · Earlier atlas export

0–1. Let the 56-second video run. It is an earlier export; the dated build counts in the deck supersede its caption. These are sequences of recorded postings, not observed travel routes. The video starts muted when this slide opens; use Play or Replay in its controls if needed. The direct link opens the MP4 separately. Use the downloaded copy if the network fails.

What one arc needs

WODEHOUSE, SIR PHILIP E.—Entered the Ceylon civil service as a writer, May, 1828; … superintendent of Honduras, 1851; … governor of British Guiana; … in 1861 was appointed governor of the Cape of Good Hope.

Printed wording	Event	Historical place	Plotting seat	Year
writer, May, 1828	writer	British Ceylon	Colombo	1828
superintendent of Honduras	superintendent	British Honduras	Belize	1851
governor of British Guiana	governor	British Guiana	Georgetown	1854
governor of the Cape of Good Hope	governor	Cape Colony	Cape Town	1861
governor of Bombay (later editions; India Office List)	governor	Bombay Presidency	Bombay	1872–77

Printed clause“governor of the Cape of Good Hope”

→

Eventgovernor · 1861

→

Checked place identifierCape Colony, a Wikidata item

→

Plotting seatCape Town: a proxy, not a residence

→

ArcBritish Guiana → Cape

1–3. The quotation is the 1867 Colonial Office List entry, which predates Bombay; the Bombay row is printed in later Colonial Office editions (1878) and in the India Office List. The full table, identifiers and trace are on step 1. “Honduras” to “British Honduras” is an interpretation, kept separately from printed wording. A capital is a plotting proxy, not evidence of residence. Wodehouse is a checked example of the cross-service match; the other bridges need review.

Two Lists, one career, mapped

Colonial Office List, 1878 · raw OCRWODEHOUSE, SIR PHILIP E., G.C.S.I. (created, 1876), K.C.B. (Creat. 1862).—Entered the Ceylon civil service as a writer, May, 1828; promoted to be assistant colonial secretary and clerk of the executive and legislative councils, Oct., 1833; district judge of Kandy, 1840; government agent for the western province, 1843; was appointed superintendent of Honduras, 1851; in Feb., 1854, governor of British Guiana; was employed in 1858 on a special mission to the government of Venezuela; and in 1861 was appointed governor of the Cape of Good Hope, and also high commissioner in South Africa. Governor of Bombay, 7th May, 1872, to 30th April, 1877.

India Office List, January 1889 · raw OCRWODEHOUSE, SIR PHILIP EDMUND, K.C.B., G.C.S.I.—Writer, Ceylon civil service, May, 1828; assistant colonial secretary, and clerk of the executive and legislative councils, Oct., 1833; district judge of Kandy, 1840; government agent for the Western Province, 1843; superintendent of Honduras, 1851; governor of British Guiana, 9th Feb., 1854; employed on a special mission to Venezuela, Feb.-May, 1858; C.B., May, 1860; governor and commander-in-chief of the Cape of Good Hope and its dependencies, and Her Majesty's high commissioner for the settling of the affairs of the territories adjacent to the eastern frontier of that colony, 29th Oct., 1861; K.C.B., July, 1862; governor of Bombay, 6th May, 1872, to 30th April, 1877; G.C.S.I., June, 1876.

One man in two Lists? The same name and a K.C.B. of 1862 in both entries link the records into one career.

The Lists disagree: Bombay from 7 May 1872 (Colonial Office) or 6 May (India Office)? Keep both readings, each with its source.

Two printed records → one linked career → arcs between dated postings, not journeys.

Animated, about 12 seconds; use ?motion or Play animation if Windows has animations off. Both texts are the raw OCR entries the pipeline read: Colonial Office List 1878 (record kgp_col1878-p447b3) and India Office List January 1889 (kgp_iol1889_jan-c2242376). The cross-service link is the project’s confidence-A bridge, evidence a shared K.C.B. of 1862. As each posting appears, the same clause lights in both Lists (blue Colonial Office, gold India Office) and lands on the map. Ceylon includes the Kandy judgeship (1840), too close to Colombo to show at this scale. The Venezuela mission is included because the atlas keeps missions as eligible postings. Both Lists print the whole career, including Bombay; the link joins two administrative records and exposes the one-day disagreement over the Bombay start date. The two entries may draw on the same official returns, so agreement is not independent corroboration. For other linked officials, check what each List adds before claiming that one shows only part of a career. Atlas link for questions: https://jimclifford.ca/col_matching/?p=kgp_col1878-p447b3 Plotting points are approximate seats on Natural Earth outlines; arcs connect recorded postings in order, not observed travel.

What the records make visible

46,926person records across two Lists

305,164career events

211,581events linked to a Wikidata place

36,431arcs for 16,287 roster records with moves (September build)

• Administrative accounts of careers, offices and honours.
• Person counts resolved within each List; their sum includes cross-service overlap.
• Seats used for plotting; arcs connect postings, not observed journeys.
• People are another matter: only 1,066 of 27,526 Colonial Office people (3.9%) link to a Wikidata person.

Whose careers does this source make easy to see? Whose does it leave out?

4–5. Ask for one response, room or chat. Official records make the administration’s categories visible, not everyone who worked for or lived under it. Grounding covers 1,066 of 27,526 CO people, about 3.9%; person links are selective, not representative. Event-to-place coverage is a different measure. Stats copied from 4 September.

Three questions

	The question	You decide
Structure	What is a record here?	What counts as an event
Clean	One person or two?	Merge, keep apart, review
Ground	Which place, which person?	Supported, unresolved, wrong

OCR text85 volumes

→

Structureentries → events

→

Cleaneditions → people

→

Groundnames → identifiers

→

Graph and atlasvisualise, then look

↺ See · Fix the rule · Rerun for every affected record

5–7. A historian owns the decisions; the agent writes scripts that apply them at scale. Today participants mark up an entry and make three decisions. We start after OCR. The published description mentions Britannica OCR tools; this session uses existing olmOCR output and does not demonstrate those tools. Your own dataset is the take-home application. Put the site and exercise links in Zoom chat.

The source

BALMER, F. E.—Dep. chief acctnt., G.P.O., E. Africa Prot., Mar., 1919; ch. acctnt., G.P.O., Kenya, Oct., 1919; ch. acctnt., Kenya-Uganda-Tanganyika post and tel. service, 1933; senr. dep. P.M.G., do., 1935.

85 volumes OCR’d, 1867–1966, including later Dominions and Commonwealth Relations Office Lists. Services of principal officers from 1867; the full Record of Services from 1883. The same person returns in many editions.

What is OCR? Optical character recognition turns a page image into text. These volumes were read by olmOCR, an open model that returns text blocks with their position on the page. OCR errors and printers’ errors look alike in the text; the scan decides.

What is “do.”? Ditto: “the same as before”. The printer saved space by pointing back to an earlier place or office. A reader resolves it without thinking; a program has to be told.

7–8. Keep OCR and source separate. A surprising date may be an OCR error or a printed error. Each record needs a volume and page reference; the packet’s source notes are not scans. The 1946 restart lacks the usual Record of Services.

Mark up Balmer

Choose entry 2. Tag the last two positions, their places and years.

Three minutes, then compare:

What does “do.” inherit?
Where should that decision be recorded?

QR code for the mark-up exercise

jimclifford.ca/imperial-careers-workshop/annotate/

8–13. Three minutes marking, one minute comparing, one minute debrief. Ask one room participant and the chat monitor for a reason. Paper users circle the corresponding words. Protect this time. The remaining entries are take-home practice.

Start with the raw OCR

I showed Claude Code the text and discussed how to structure it.

ch. acctnt., Kenya-Uganda-Tanganyika post and tel. service, 1933; senr. dep. P.M.G., do., 1935.

Could Python do this on its own? What would the rules need to know?

Read a sample together. Agree what counts as a record and what fields to keep.

edition 1936 · printed page 614
OCR page index 832 · block 11
bbox [142, 2184, 827, 2334] · category text

Balmer’s original OCR block: every record keeps its way back to the page

What is Claude Code? Anthropic’s coding agent. It runs on your own computer, reads the files you point it to, and writes and runs Python with your permission. You describe the task in plain language and judge the results; it reports what it did.

It is a commercial service, paid through a Claude subscription or per use. Here it built the pipeline; it did not read the 65,000 career records.

13–14. Start with the actual working process: I opened the raw OCR with Claude Code and discussed whether a Python parser could do the job. The displayed excerpt is copied directly from the 1936 raw OCR block, before abbreviation expansion. The saved file includes the original bounding box, OCR page index 832, block index 11 and printed page 614. This is a walkthrough of the process, not a recovered chat transcript. Use the Balmer exercise they just completed: they already know that “do.” refers back to a place. Ask what the code would need to preserve to reach the same reading. Claude Code helped inspect the source, discuss the approach and write the scripts; the historian decided what the fields meant.

Real rules: mostly right, quietly wrong

Our real rule-based parser: 258 lines of Python, run on every printed biography in 66 editions. It splits a career at semicolons, knows months, abbreviations and ditto marks, and refuses an entry whole when a dated clause will not parse.

EntryPrinted clauseThe rules returnedVerdict Balmer, 1936ch. acctnt., G.P.O., Kenya, Oct., 1919place: Kenya · 1919✓ the regular shape works Balmer, 1936senr. dep. P.M.G., do., 1935place: Dominions Office · 1935✗ two rules collide Connolly, 1940agt. gen. in London for Malta, 1930place: Western Australia · 1930✗ accepted, and wrong Wodehouse, 1867and in 1861 was appointed governor of the Cape of Good Hopeentry refused → Qwen: 9 events, each placed✓ refused, not guessed

78% of 202,100 printed biographies parsed by rules

61% → 96% rules share, 1888 vs 1960 editions

3,755 places dropped mid-clause, recovered later

Rules are fast and exact on regular entries. Each new rule can collide with another, and the dangerous errors are the confident ones.

14–15. These are the project’s real rule-based parser and its saved output, not a mock-up (outputs/structure/rules-parser-examples.json; col_matching col_match/services/rules_parse.py, data/volume/col*/bios.jsonl). The parser is not naive: it splits at semicolons because the printer did, and it has a month list, an abbreviation gazetteer, ditto handling and all-or-nothing refusal. It parsed 157,108 of 202,100 printed Colonial Office biographies across 66 editions; its share rises from 61% (1888) to 96% (1960) as house style became more regular. The refusal works: Wodehouse’s 1867 prose entry was refused and Qwen read all nine events. The problem is what the rules accept. Balmer’s “do.” became Dominions Office, because the abbreviation “do” fires before the ditto rule; Connolly’s Malta posting was accepted with Western Australia carried forward. A later pass recovered 3,755 places the rules had dropped because the colony sat mid-clause. Ask: what rule would you add next, and what would it break? This is the edition-reading stage; the next slides show why the final records were read by the model.

Too irregular for regular expressions alone

A familiar historical problem: semi-structured data.

The fields recur. The wording and relationships vary.

Source wording	What needs interpretation
“do., 1935”	Which preceding place does this posting inherit?
“B. Honduras, Apl., 1898”	An appointment date; “B.” belongs to the place name
Trimen’s posts, 1867–1875 and 1869–1879	Two concurrent jobs, not one sequence

Finding a year is easier than deciding what it dates.

What is a regular expression? A pattern that matches text by its shape. 1[89]\d\d finds every year from 1800 to 1999 in an entry. It cannot say which posting the year belongs to, or that “do.” points back three clauses.

15. This is a source problem historians know well: recurring headings, names, dates and offices without a uniform sentence structure. OCR can add variation, but even perfectly transcribed entries contain elision, abbreviations, compound clauses and concurrent appointments. Regular expressions recognise specified patterns; increasingly elaborate rules did not cover the collection reliably. Python is still useful for segmentation, regular cases and checks. Explain the three rows: Balmer inherits the preceding service’s place; Blakely’s “B. Honduras” is British Honduras and April 1898 dates his clerkship; Trimen’s lecturer and British Museum posts overlap. The third example is in the saved Trimen source pack. The conclusion of our discussion was that Python rules alone were insufficient for this corpus. The hard part is binding the right position, place and date, not simply detecting a number.

Develop the hybrid with Claude Code

Agree the fields → try examples → decide which work belongs in each stage.

Stage 1 · reading every edition, about 200,000 printed biographies

Python rulesParse the regular entries; refuse the rest whole.

→

QwenReads the refused entries.

→

LinkEntries joined into people across editions.

Stage 2 · the final records, 65,239 careers

PythonExpands known abbreviations; keeps the rest as printed.

→

Qwen reads every careerOne entry in, one JSON record out: position, place and dates bound across ditto, elided and concurrent clauses.

→

Python checksDrops any place or year not in the source; applies logged fixes.

What is Qwen? A family of open-weight language models from Alibaba’s Qwen team. Open weight means the trained model can be downloaded and run on your own hardware, or reached through a hosted service. The structuring runs used Qwen3.6-35B-A3B: 35 billion parameters in all, about 3 billion active for each word-piece, so one GPU can run it quickly.

Claude Code helped build the pipeline. Qwen read the entries.

15–16. Walk through the design as a division of work developed with Claude Code, in two stages. Stage 1 read every printed edition to link entries into people: Python rules parsed the regular entries and refused the rest whole, and Qwen read the refused ones (78% rules overall). Stage 2 produced the final structured records: after the rules’ silent errors (previous slide), Qwen read every one of the 65,239 deduplicated careers, after Python expanded known abbreviations; Python then checked every place and year against the source and applied logged corrections. One event has position, place, start/end years and an inheritance flag. Compare sample outputs with the OCR before increasing the run size. Claude Code was the interactive coding environment; Qwen3.6-35B was the model called by the corpus-reading script. The saved in-session runbook records how the reference examples were structured and validated.

Three ways to run Qwen

The script sends each entry to a web address that speaks the OpenAI request format.
Change one address to change where the model runs.

Hosted service

OpenRouter

A broker for many models, open and commercial. Pay per token; no hardware.

Used in June to compare several open models on the same 40 entries, scored against records Claude had structured in-session. Qwen3.6 came closest.

Desktop GPU

NVIDIA DGX Spark

A desktop machine with 128 GB of memory shared with its GPU. vLLM serves the model on the local network.

India Office List: 30,496 entries in 8.24 hours overnight. No per-request bill.

Research cluster

Alliance: Narval, Nibi

Canada’s national research computing. Submit a batch job; it waits in a queue, starts the model, works through a file.

Qwen3.8-27B judged 3,539 identity pairs in 35 minutes on one A100 GPU.

What is vLLM? Open-source software that loads a model onto a GPU and answers requests in the same format as the commercial services, so the same script works locally, on a cluster or through OpenRouter.

16–17. One slide on infrastructure, because participants ask how to run the model. The script did not change between routes: it posts the system prompt and one entry to an OpenAI-compatible endpoint, and the base URL selects the service. The June bake-off (19 June, 40 entries against the in-session Opus reference set, reasoning off, temperature zero) ran several open models through OpenRouter; Qwen3.6-35B-A3B had the best event precision, place and honours scores. That reference set is provisional, not a gold standard. Production ran the same model on a DGX Spark desktop through vLLM, with no tunnel to depend on. The cluster example is later identity work, not structuring: Narval job 2430423, Qwen3.8-27B, one A100-40GB, 1.7 pairs a second. Cluster compute nodes have no internet, so the job serves the model inside the allocation. Any change of model or serving setup needs the sample audit again.

The prompt is the decisions

Extract ONLY what is printed; never invent or infer facts.

events: one object per distinct posting. EXPLODE compound clauses.

Place rule, summarised: keep printed wording; flag inheritance and stop at a stated transfer.

birth_year: only if printed. NEVER infer from the earliest career date.

What is a system prompt? The standing instructions sent with each of about 65,000 career records. The entry itself follows as the user message. Every rule here exists because an earlier run broke it.

17–19. This is what the discussion became: explicit instructions for the model inside the pipeline. The first, second and fourth quotations are from the saved prompt; the place line summarises its inheritance rule, including the stop at a stated transfer. Link to the exact prompt on step 3. Ask: would “defeated in an election” count as an office held? Leave that distinction in mind for Brewster.

Read records against the source

Examples from the saved extraction audit, before rule fixes

Source	Earlier output	Reading to preserve
“do., 1935”	place: “do.”	Inherit the preceding place
“Apl., 1898”	birth_year: 1898	Date of the appointment
“KT. BACH. (1920)”	two honours	One knighthood

Sampleevery tenth record, then spread across the corpus

→

Read against the sourcerecord beside the printed entry

→

Sort findingsone-off, or a recurring pattern?

→

Logged ruleor individual review

Batch run: 30,496 entries in 8.24 hours.

19–20. The model’s output still needs to be read against the source. These are historical extraction outputs used to teach that reading, not a report of current defects. All three audits were read by a model; the displayed examples were checked by hand. The detailed sample counts remain on step 3 for questions; they do not measure finished-graph accuracy. Ask participants to explain one reading before moving to the Balmer replay. Some findings need individual review rather than a general rule. The stage-2 run read 30,496 India Office careers in 8.24 hours, 25–26 June, with 16 workers, reasoning off and temperature zero. Put this scale result after the design and review process. If asked: 55 entries failed JSON validation, a format count rather than a count of inaccurate records.

Replay a past correction

Balmer’s final event · saved extraction output

Beforeplace: “do.”
place_inherited: true

→

RuleA literal ditto, and the previous event has a resolved place → inherit that place and log the reason

→

Afterplace: “Kenya-Uganda-Tanganyika”
place_inherited: true

3 saved records · 12 events · 1 place corrected

• Other events and fields unchanged.
• Second run changes nothing.
• No preceding resolved place? Leave it for review.

Saved before-and-after records

20–23. Use the saved replay; run python3 scripts/demo_ditto_fix.py only if time allows. No model or network is needed. Introduce this as a replay of an earlier correction using saved records. It is not a rerun of col_matching. The other eleven events and the saved Blakely and Connolly fields are unchanged: this demonstrates the scope of the ditto rule. The missing-predecessor checks are constructed tests, not historical records. Ask what else must be checked before applying this to the corpus: affected records, boundary cases, a fresh sample afterwards.

Balmer’s ditto, resolved

BALMER, F. E.—Dep. chief acctnt., G.P.O., E. Africa Prot., Mar., 1919; ch. acctnt., G.P.O., Kenya, Oct., 1919; ch. acctnt., Kenya-Uganda-Tanganyika post and tel. service, 1933; senr. dep. P.M.G., do., 1935.

do. = ditto, “the same as before”: it points back to the last place named.

{ "position": "senior deputy P.M.G.",
  "place": "do.","Kenya-Uganda-Tanganyika",
  "year_start": 1935, "place_inherited": true }
+ change logged: “Printed ditto; previous event has a place.”

A reader resolves it at a glance. A program has to be told, and the change is logged.

Animated, about 8 seconds; it shows the correction from the replay slide before it. The record lines are copied from the saved before/after file (event 4); only the place changes, with the logged reason.

Clean: forty editions, one person

A missed link leaves a fragmented career.
A false merge can silently corrupt later analysis.

Ties resolve to no merge.

199,099printed entries, Colonial Office List

→

Strong rulessame name and birth year, or a shared dated event

→

Clusterssame forename, same surname ending, so OCR variants meet

→

Dossiersa model argues each hard case; review queue for the rest

→

27,526person records, with a ledger of reasons

Dossier Every version of every biography in a cluster, laid out for one decision: births, schools, honours, the full dated career.

Ledger An append-only file of merge decisions, each with its reason, so any one can be revisited and reversed.

23–25. 199,099 printed CO entry instances resolved into 27,526 person records. The ledger makes decisions revisitable; do not call false merges irreversible. Explain a dossier as the source evidence for both candidates, gathered in one place.

Decide before the answer

One edition

ABERY, Joseph

M.B.C.M.

1867 · Ceylon · assistant colonial surgeon

Six editions

CARBERY, Joseph

M.B.C.M.

1867 · Ceylon · assistant colonial surgeon

Merge, keep apart, or review? What evidence decides it?

The ledger merges them: consistent name and qualification, an identical dated appointment, and a plausible OCR loss. Preserve both entries and the reason.

25–28. Give 30 seconds to decide; room and Zoom use the same three words. Then reveal. Qualification is not an honour. Checking the scan for the lost letters would strengthen the decision. A common first name alone is insufficient.

Keep apart; keep the evidence

Born 1836

A’BECKETT, Thomas

Lincoln’s Inn

Victorian judge · knighted 1909

Born 1898

BECKETT, Thomas Rumbold

Judd School

Nigerian telegraph engineer

Different schools and careers, not only different birth years. Keep apart.

The merge rules, in brief Merge only on positive evidence: an overlapping dated event, a shared honour with its year, near-identical education, or an OCR variant with a consistent career. Birth year alone never splits: it is often misread. Different spelled forenames keep people apart.

28–30. A birth-year discrepancy alone is not a universal veto: it may be OCR. Here the whole dossiers differ. The supporting page has more clusters; do not run all four aloud.

Ground: retrieve, read, decide

Every plotted posting needs a place.
Every shared person identifier asserts an identity.

Mention“Victoria, 1912”

→

Retrievecandidate items from a search tool

→

Readtype, dates, location against the career

→

Decide and recordsupported · unresolved · wrong, with evidence and date

No supported match? Keep the local identifier.

What is Wikidata? A QID? A free, shared knowledge base. Each item has an identifier: Q plus a number. Q1800510 is Philip Wodehouse. Linking to it lets other projects join their data to ours.

What is MCP? Model Context Protocol: a standard way to hand a model a tool, such as a Wikidata search. The answer comes back as retrieved evidence, not from the model’s memory.

30–31. An identifier’s existence is not evidence that it belongs to this mention. Introduce MCP simply as a lookup tool the agent can use; the protocol is not today’s lesson.

One entry, three kinds of link

LOGAN, Sir William Marston, K.B.E. (1944) … —b. 1889; ed. Bristol and Brasenose Coll., Oxford …; ch. sec., N. Rhod., 1937; ag. gov., Dec., 1937 …; gov. and c.-in-c., Seychelles, 1942; retd., 1947 …

Places · WikidataMCP search “N. Rhod.” → Northern Rhodesia “Seychelles” → Seychelles Read the candidate’s type, country and dates; prefer the colonial-era entity. 131,343 of 152,132 placed events (86%)

Person · QLever query name · b. 1889 · Northern Rhodesia → William Marston Logan Q108649219. Surname, given names, birth year and office all agree; died 1968, after his last listing. 1,066 of 27,526 people (3.9%)

School · WikidataMCP search “Brasenose Coll., Oxford” → Brasenose College Q899030. “Bristol” stays unlinked: a city, not a named school. 11,056 of 15,775 education links (70%)

Places are many and easy to check. People need more evidence, so most stay with a local identifier.

30–31. One printed entry, three kinds of link, each with its own method and its own evidence threshold. Logan’s entry is from the 1948 Colonial Office List. Places: the agent searched WikidataMCP and read each candidate’s statements; the place strings were first collected and normalised (5,376 distinct strings in the Colonial Office cache), colonies came from a curated list of 740, and towns went through MCP batches. People: 1,066 of 27,526 Colonial Office person records are linked, about 3.9 per cent; the rest keep local identifiers. Schools: education strings were split into institutions (rules for the Colonial Office List, Qwen for the India Office List) and searched; bare multi-referent names such as “St. John’s College” were quarantined. The List calls Logan acting governor of Northern Rhodesia; Wikidata lists the governorship as a position held. Counts from col_matching graph_stage3 files.

WikidataMCP: search by meaning, then read

Printed string“Penang, S. Sttlnts.”

→

Normalised query“Penang Straits Settlements”

→

search_itemscandidates ranked by meaning, not spelling

→

get_statementsinstance of · country · located in · coordinates

→

Cache the decisionwith its evidence, so the graph rebuilds without new lookups

Rules the agent followed

Prefer the colonial-era entity: British Ceylon, not Sri Lanka.

Leave compounds and ambiguous initialisms (“Ken. and Uga.”, “N.S.”) ungrounded.

Never write an identifier from memory.

What the checks caught

A later check of every identifier caught five invented by parallel agents: Dehra, Ayrshire, Rohilkund, Neemuch, Narsinghpur.

Fuzzy reuse flipped N./S. and E./W.; only exact spelling variants were folded (166).

What is vector search? Each Wikidata item is stored as a list of numbers that captures what its description means, so a query finds items close in meaning, not only in spelling. It still returns candidates, not answers: on 11 October “Penang Straits Settlements” put the whole colony first, the modern Malaysian state eighth and Penang Island 24th. Reading the statements decides.

31–32. The agent (Claude Code, in session, with a place-disambiguation skill) made the MCP calls; Python only prepared work lists and recorded answers. Endpoint wd-mcp.wmcloud.org, rate limited, so results were cached aggressively. Every QID was later re-checked with batched SPARQL or wbgetentities; the five invented identifiers came from parallel sub-agents in the India Office tail batches (col_matching commit 2d457a5). The N./S. flips came from fuzzy matching in the reuse pass, which is why only exact-token variants were accepted. Sources: docs/GROUNDING_NEXT_100.md, col_match/kg/ground.py.

QLever: ask exactly when you can

Who held a position named “Governor … Northern Rhodesia”? Give their birth dates.

SELECT ?person ?personLabel ?dob WHERE { ?pos rdfs:label ?posLabel . FILTER(CONTAINS(?posLabel, "Northern Rhodesia")) FILTER(CONTAINS(?posLabel, "Governor")) ?person wdt:P39 ?pos ; wdt:P31 wd:Q5 . OPTIONAL { ?person wdt:P569 ?dob } }

1 Birth within two years of the List’s 1889

2 Same surname and given names: LOGAN, William Marston

3 Death check: died 1968; last listed 1948 ✓

Wikidata holders returnedborn

William Marston Logan1889

Sir John Maybin1889

John Waddington1890

Hubert Winthrop Young1885

Ronald Storrs1881

Herbert Stanley1872

James Crawford Maxwell1869

1,095 senior officials without a link

198 colony names queried

124 people linked this way

What are SPARQL and QLever? SPARQL is the query language for Wikidata: ask for every item matching a pattern of statements, such as “held this office and was born in…”. QLever, from the University of Freiburg, is a fast public engine that runs such queries over all of Wikidata.

32–33. Use QLever when the entry gives enough structured evidence to ask an exact question: an office, a jurisdiction and a birth year. This is a real result: the project’s query for Northern Rhodesia governorships returned these seven people. Two were born in 1889, so the birth year alone cannot decide; name, birth year and office together do. Rule (kg_ground_persons_l2_match.py): equal surname, compatible given names, no link if birth years differ by more than two, exactly one surviving holder, and a guard against two people claiming one QID. Then verify_person_deaths.py drops any match where the List shows the person more than a year after Wikidata’s death date (21 dropped). The List calls Logan acting governor of Northern Rhodesia (1937–41) and governor of the Seychelles (1942); Wikidata lists both governorships. The query on the slide is simplified from kg_ground_persons_l2.py, which also accepts Secretary, Administrator and Chief Justice titles. Most of the 1,066 linked people came from an earlier pass against a harvested set of Wikidata officials with the same birth-year rule.

What reading the corpus costs

Measured on a random sample of 2,000 India Office entries and 2,000 Qwen records:

~640tokens read per entry: 470 of prompt, about 150 of entry

~330tokens written per entry: the JSON record

30,496India Office List entries

Route	Price	30,496 entries
OpenRouter, Qwen3.8-27B, 8 Oct	$0.0239 read / $2.175 written per million tokens (50% promotion)	≈ $22
Same model, regular price	about twice the promotion	≈ $45
DGX Spark desktop, Qwen3.6 (as run)	the machine and its electricity	8.24 hours, no bill per request

Almost all of the hosted cost is the writing: the record the model returns, not the text it reads.

What is a token? The unit models read and write in, roughly three-quarters of a word. Services charge per million tokens, with separate prices for reading (input) and writing (output).

38–40. Worked estimate, not an invoice. Token counts use the Qwen tokenizer on a random 2,000 India Office entry instances and 2,000 saved Qwen records; input is the 471-token system prompt plus the entry and a small chat-template allowance. 30,496 × 640 ≈ 19.5 million tokens read ≈ $0.47; 30,496 × 330 ≈ 10 million written ≈ $22. Add retries (four allowed) and failures. The OpenRouter price was 50% off on 8 October 2026; check the current price on the day. Qwen3.8-27B was not the structuring model: switching models needs the sample audit again. For comparison, the June bake-off estimated about $13 for the Colonial Office List on Qwen3.6 through OpenRouter at June prices. The desktop has no per-request bill, but the machine, power and the historian’s time are real costs.

A past repair: which Victoria?

Brewster’s entry: “defeated by Sir Richard McBride, in Victoria, 1912.”

An earlier atlas placed a British Columbia politician in Australia.

Colony of Victoria, AustraliaVictoria, British ColumbiaVictoria, Hong KongVictoria, SeychellesVictoria, Labuan

One printed word; several imperial places with that name.

BeforeOne “Victoria” lookup, reused for every entry that printed the word → Australia

→

Completed fixResolve the place per person, from the surrounding career: McBride, the legislative assembly, British Columbia

→

AfterVictoria, British Columbia; every affected entry re-emitted

40–42. State that this is a completed repair before discussing the old result. Explain the per-person resolver and the identifier-key bug it exposed. Both corpora were re-emitted. Ask what evidence selects British Columbia: the surrounding career clauses. The supporting page retains South Africa and Baluchistan as further historical examples.

Which Victoria?

Brewster: “defeated by Sir Richard McBride, in Victoria, 1912.”

Resolve the place per person, from the surrounding career. Every affected entry is re-emitted.

Animated, about 8 seconds; it shows the repair on the Victoria slide before it. Five real imperial places named Victoria; the earlier atlas reused one match (the Australian colony) for every entry. The repair resolves per person from the career context. Points are approximate seats on Natural Earth outlines.

A past repair: what counts as an event?

Source	Earlier graph, after the place fix
Defeated, Victoria, 1912	Legislative assembly member for Victoria, 1912–1916

Would you accept this office-held assertion?

A defeat does not establish an appointment. Repaired: preserve the defeat as an event; remove the office-held assertion. Check other defeat clauses.

A plausible map can conceal a misread source.

42–44. The table is a saved stage in the repair, before the event correction. Pause for a room/chat verdict before revealing the completed repair. The public graph now types this as an electoral defeat and removes the held-role edge. The same check catches other explicit electoral-defeat events. Use this to teach the difference between recording an event and asserting an office held.

What a map cannot check

The map shows

What looks wrong

A politician in the wrong hemisphere. A jump no career could make. Two nodes with one name.

The map hides

What is missing

A missed link leaves two plausible short careers. An ungrounded posting simply vanishes. A false merge can look like an ordinary career.

Use a labelled sample, including the cases the pipeline rejected.

Check both the repaired cases and other records the rule changed.

44–46. Explain why source sampling complements visual inspection: a missing record may produce no visible anomaly. Include accepted and rejected decisions in the review design. Keep the historical audit figures on the supporting pages for questions; focus here on how participants would check their own data.

Three decisions, with reasons

Work from the saved teaching cases.

R1 · Blakely: accept the record?
R3 · Abery / Carbery: accept the merge?
R8 · Harris: accept a link with a disputed birth year?

Decide. Give the evidence. Say what you would check next.

Six minutes working; four minutes discussing R8.

QR code for the three-decision exercise

jimclifford.ca/imperial-careers-workshop/review/

46–56. Introduce these as saved proposals for practising decisions, not a live queue of atlas errors. The site displays the core cases first; the other five are further practice. Do not reveal the Harris answer until participants have decided. Zoom users select R8 and copy its decision and reason. Ask what supports the identity and what remains uncertain about the date. The matching biography supports joining the records despite a possible OCR error; preserve the disputed readings. Separate identity confidence from permission to change a source field. Printed handout page 2 has the same three decisions.

Your first twenty records

• Show the coding agent your raw text, including a hard case.
• Discuss what Python rules can handle and what needs interpretation.
• Agree a schema and review two examples against the source.
• Build the hybrid; run a small sample and record corrections.
• Rerun, check what changed, then measure a larger run.

Write one sentence: what counts as a record in your collection?

56–60. One minute for this sentence; invite one response and questions. The take-home prompt and cost worksheet are on step 7. Close by pointing to the saved prompt, replay and exercises, not by listing every tool again.

Keep the decision with the evidence

Structure · Clean · Ground

Build. Extract. Visualise. See. Fix. Rerun.

QR code for the workshop site

jimclifford.ca/imperial-careers-workshop
Exercises, worked answers, prompt, replay and glossary.