Ground

Minutes 30–40. Which place, which person, in the shared vocabulary? A live lookup, and a chatbot asked to do the same from memory.

  • grounding: linking a name in the source to a shared identifier for the thing it names.
  • QID: a Wikidata identifier: Q plus a number, such as Q1800510.
  • vector search: search by meaning rather than by matching letters; the WikidataMCP server offers it over Wikidata.
  • MCP: a standard way of giving a model a tool it can call, so answers come from retrieval rather than memory.
  • gate: a rule that accepts a link only when the evidence leaves no plausible alternative, and otherwise leaves it unlinked.
  • surface: a place name exactly as printed, before any decision about what it refers to.
  • cache: the saved results of earlier lookups, keyed by the printed name.
  • SPARQL: the query language for Wikidata; asks for every item matching a pattern of statements.
  • QLever: a fast public engine that runs SPARQL queries over all of Wikidata.
  • API: a way for a program to send a request to a service and get an answer back; here, to a model or to Wikidata.

All terms: glossary.

Why ground at all

Every arc on the atlas needs a latitude and longitude for a place named in a Victorian reference book: the Colony of Victoria, the Central Provinces, British Guiana, a princely state. Modern gazetteers do not have them or mislocate them. Wikidata has an item for each, with coordinates or a link to the capital that has them. For accepted place links, a query can retrieve coordinates or a capital. Unresolved names remain unplaced, and the choice of a historical entity and a period-appropriate seat still needs checking.

The second reason is composition. A historian of the Cape and a historian of Bombay who both ground to Q1800510 have joined their datasets without meeting. The longer argument is Jim’s; the short version is: ground to Wikidata, do not host your scholarship there.

Two ways to get a QID

Retrieve it. For places and schools the pipeline uses the WikidataMCP server’s vector search, which matches meaning rather than spelling, then reads the item’s statements, then applies verification gates (type, dates, place) before accepting. For people with enough structured evidence it asks Wikidata an exact question with a SPARQL query (below). Every accepted link keeps its evidence and lookup date. The Colonial Office List graph has 131,343 events grounded to a place and 1,066 of its 27,526 person records linked (about 3.9%). The gates aim to avoid false matches; that is a design aim, not a demonstrated zero-error rate. Low coverage matters: the linked people are not a representative sample of the roster.

Ask a chatbot. Paste the biography into a model with no tools and ask for the QID. It will give one. Sometimes it is right.

From structured records to identifiers

The structured JSON records name places, schools and people as printed. Each kind went through its own route, with its own evidence threshold. Logan’s entry in the 1948 Colonial Office List shows all three: “ch. sec., N. Rhod., 1937; … gov. and c.-in-c., Seychelles, 1942”, “ed. Bristol and Brasenose Coll., Oxford”, “b. 1889”.

Places.

  1. Collect every distinct place string and normalise abbreviations into a search query: “Penang, S. Sttlnts.” becomes “Penang Straits Settlements”. The Colonial Office cache holds 5,376 distinct strings. Offices, wars and months are set aside, never grounded.
  2. Match colonies against a curated list of 740 colonies with their dates before any search.
  3. For towns and districts, the coding agent searched WikidataMCP in batches and read each candidate’s statements: instance of, country, located in, coordinates. Rules: prefer the colonial-era entity (British Ceylon, not Sri Lanka); leave compounds and ambiguous initialisms ungrounded; never write an identifier from memory.
  4. Fold exact spelling variants onto identifiers already found (166 in one Colonial Office pass). Fuzzy matching was rejected because it flipped N./S. and E./W.
  5. Check every identifier afterwards with a batched query. In the India Office tail batches this caught five identifiers invented by parallel agents (Dehra, Ayrshire, Rohilkund, Neemuch, Narsinghpur).
  6. Resolve ambiguous strings per person from the rest of the career (Victoria, S.A., W.A., Perth, Kingston), and apply date rules: no Union of South Africa before 1910; capitals by period.

Result: 131,343 of 152,132 placed Colonial Office events link to a Wikidata place (86%). The search returns candidates, not answers. On 11 October “Penang Straits Settlements” returned the whole colony first, the modern Malaysian state eighth and Penang Island 24th.

People. A person link needs more than a name. The first pass compared the graph with a harvested set of Wikidata colonial officials: same surname and first initial, no conflicting given names, and no link if the birth years differ by more than two. Linked QIDs were then re-checked against Wikidata (human, birth date, offices, occupation), and colony-only matches without office support were dropped. A second pass asked QLever, a fast public SPARQL engine, for everyone who held a governorship, colonial secretaryship or chief justiceship named for each of 198 colonies, with birth dates, for 1,095 senior officials still without a link. The Northern Rhodesia query returned seven governors; two were born in 1889, so the birth year alone could not decide. Surname, given names, birth year and office together matched William Marston Logan (Q108649219). The second pass linked 124 people. A final check dropped 21 links where the List shows a person more than a year after Wikidata’s date of death. Total: 1,066 of 27,526 (3.9%). Person links for the India Office List are deferred.

Schools. Education strings were split into institutions, by rules for the Colonial Office List and by Qwen for the India Office List, where rules reached only about 56% of rows. Each institution was searched with WikidataMCP, checked for type, country, coordinates and same-name twins, and recorded as a Wikidata link, a fold onto an existing link, a project identifier, or quarantined as ambiguous (a bare “St. John’s College”). Logan’s “Brasenose Coll., Oxford” links to Q899030; “Bristol”, a city rather than a named school, stays unlinked. Result: 11,056 of 15,775 Colonial Office education links (70%).

Sources: Imperial Careers docs/GROUNDING_NEXT_100.md, docs/GROUND_INSTITUTIONS.md, kg_ground_persons.py, kg_ground_persons_l2.py, verify_person_deaths.py, and the graph_stage3 files.

Live: the lookup

Read the whole biography, allowing for OCR errors in names and dates. When education, offices, places and the career sequence agree, link the identity despite an isolated discrepancy. Preserve the conflicting readings with their sources and flag the field for review. Keep people apart when the careers themselves conflict; a single misread digit should not fragment an otherwise matching biography.

To try it yourself with a coding agent connected to the WikidataMCP server (the public web endpoint is heavily rate-limited), run these three searches:

  1. Search “Henry Trimen botanist Ceylon Peradeniya”. Read the top hit’s statements: born 1843, died 1896, botanist. Compare with the entry: director from 1880, M.B., F.R.S. Supported.
  2. Search “Ferdinando Hamlyn Price” with the career words. The top hit is Q123279571, “Ferdinando Hamblyn Price”, alias Hamlyn, born 1855, died 1942, described only in Tamil as a Sri Lankan politician, no career statements at all. The entry says mayor of Colombo three times. Supported, on the name, the dates and Colombo, and the pipeline’s gate had rejected it because the spelling differed.
  3. Search “Elwood D’Arcy Tibbits Antigua”. Nothing plausible in the top fifty. Unresolved. Keep the local record and the search that was run.

A side experiment: twenty entries, three workflows

Before trusting a model with identifiers, we gave the same twenty entries to a chatbot without tools and to two models with web retrieval. The chatbot supplied sixteen person identifiers and none was right; the retrieval runs got sixteen of sixteen but took twenty to twenty-seven minutes. The full write-up, with the scoring and what each workflow costs in time, is on its own page: Side experiment: twenty entries, three workflows.

Three questions to keep separate

  1. Does the identifier resolve to what was retrieved? A Q-number from a model’s memory is not a lookup.
  2. Is that entity the one in this source? Name, type, dates, place, activity. A real item can be the wrong person. Price’s item has the right dates and no career; the match rests on a rare name and Colombo.
  3. What does the link assert? That this entry is that man. Not every statement on the item. Not that “Ceylon” in 1880 is the modern state.

Places are harder than people

The surface “Victoria” occurs in the corpus as the Australian colony, the capital of British Columbia, an electoral district, a gaol in Hong Kong, a district in the Cape, a county in Mauritius, and a town in the Seychelles. The grounding cache (rows) shows what that took: the bare surface grounded to the colony by default, context rules for officials whose other postings are in British Columbia, manual re-grounding for the Cape and Seychelles variants, and six surfaces deliberately left ungrounded because one QID would be wrong for someone.

You decide: supported, unresolved, or wrong, for each link that matters, and the gate that decides the rest. The cases the gate gets wrong are the subject of the next step.

Next: The map finds the errors.