Cleaning and Structuring Historical Data: A Working Session
AI and History Conference, Johns Hopkins University. Thursday 15 October 2026, 1:30–2:30 p.m., Room 110, SNF Agora Institute. Streamed on Zoom.
Source and data: github.com/jburnford/imperial-careers-workshop. The dataset the hour is built on is published as Imperial Careers (CC0). Counts on these pages come from that repository’s kg_stats.json and the atlas meta.json of 4 September 2026 and are reproduced in outputs/atlas/.
One dataset, start to finish
Two printed reference works, the Colonial Office List (published 1862–1966; the project used editions from 1867) and the India Office List (1886–1947), carry a terse biography for every senior official: born, schooled, every posting, every honour. The same man recurs in forty annual editions. The OCR garbles a name here and there. Two men who share a surname threaten to collapse into one career.
Over a few months, one historian with a coding agent turned those volumes into a knowledge graph and an animated atlas of imperial careers:
| Measure | Colonial Office List | India Office List |
|---|---|---|
| Person records within each List | 27,526 | 19,400 |
| Career events | 180,470 | 124,694 |
| Events located to a Wikidata place | 131,343 | 80,238 |
The hour is about how, and about where in that process a historian has to decide something. The two person counts are resolved within each List; their sum is a roster count, not a count of unique people across both services.
By the end you will have marked up an entry, judged a proposed record and an identity match, and decided how to preserve a disagreement between sources. We work on a shared example in a browser; the final step takes the method to your own material.
Three questions
Technical terms are explained where they first appear on each page, and in the glossary.
A historian turning a text into data asks three questions of it, and each has a decision the historian must own. Models and scripts carry those decisions across the collection; their output still needs review.
| Step | The question | The model does | You decide |
|---|---|---|---|
| Structure | What is a record here? | Proposes the fields, fills them for about 65,000 career records, reports what it could not parse | What counts as an event; whether the record changed the meaning |
| Clean | Is this one person or two? | Clusters the candidates, argues each case, keeps the evidence | Merge, keep apart, or leave for review |
| Ground | Which place, which person, in the shared vocabulary? | Retrieves candidates with evidence | Supported, unresolved, or wrong |
The second half of the method
Extraction is half of it. The other half is looking at the result in a form where errors are visible. The atlas was built to answer a research question, and it turned out to be the best error detector the project had. A British Columbia premier appearing in Australia, a cluster of South African postings landing at Harare, two nodes both called Baluchistan: each was seen on the map, traced to a rule, fixed in the pipeline, and re-emitted for everyone it affected. Build, extract, visualise, see, fix, rerun. That loop is what the hour demonstrates.
The hour
| Minutes | Stage | Focus |
|---|---|---|
| 0–7 | The atlas | The video; then one man’s entry turned into identifiers, coordinates and arcs, and the corrections carried through to the rebuilt atlas |
| 7–13 | The source | One printed entry. Mark up what you would extract |
| 13–23 | Structure | Raw OCR, the discussion with Claude Code, why Python rules alone were insufficient, and the hybrid workflow |
| 23–30 | Clean | Forty editions, one person: the merge rules and the cases that need a historian |
| 30–40 | Ground | A live Wikidata lookup, and a chatbot asked to do the same from memory |
| 40–46 | The map finds the errors | Brewster: the wrong place, then an unsupported appointment |
| 46–56 | Exercise | Three decisions with reasons; five further cases to take home |
| 56–60 | Your own data | Taking the three questions to another collection |
What you need
A browser. The decision exercise and the mark-up exercise run from saved results and need no account or key. Remote participants have the same pages; each exercise has a button that copies a one-line summary of your work for the Zoom chat. There is a two-page printable handout with Balmer and the same three decisions for the room.
Further
Worked answers, a glossary of the terms used, running it yourself, and the saved outputs in outputs/: the extraction prompt, the audit of a structured sample, the duplicate-resolution rules and a sample of clusters, the place-grounding cache rows behind the errors, and the project review that found them.