Structure
Minutes 13–23. From raw OCR to a hybrid workflow, developed with Claude Code.
- JSON: a plain-text format for a record: named fields in braces, lists in brackets (see the glossary for Trimen’s).
- prompt: the written instructions sent to the model with each entry.
- regex grammar: pattern rules that match text by its shape, written in Python; used to read every printed edition.
- validator: Python that checks every place and year the model wrote against the source and drops what is not there.
- post-normaliser: Python that applies the systematic fixes found by the audit, logging each change.
- vLLM: an open-source server that runs an open model on a GPU and answers like a commercial API.
- GPU: the graphics card the model runs on; the model has to fit in its memory.
- token: the unit a model reads and writes in, roughly three-quarters of a word.
- reasoning mode: a setting where the model writes out its thinking before answering; off here, because it spent the budget and returned nothing.
- frontier model / open model: frontier refers to leading capability; open refers here to downloadable model weights. These are different properties, not opposites. The project used a commercial coding agent and an open model inside the pipeline.
- deterministic: done by fixed rules, so the same input always gives the same output; no model involved.
All terms: glossary.
Start by showing Claude Code the raw OCR
I began by showing Claude Code the raw OCR and discussing whether Python could structure it on its own. We read examples, discussed what a record should contain, and worked out where explicit rules would be reliable and where the text needed interpretation.
Balmer’s original OCR block, before abbreviation expansion:
BALMER, F. E.—Dep. chief acctnt., G.P.O., E. Africa Prot., Mar., 1919; ch. acctnt., G.P.O., Kenya, Oct., 1919; ch. acctnt., Kenya-Uganda-Tanganyika post and tel. service, 1933; senr. dep. P.M.G., do., 1935.
The saved raw block retains its bounding box and source location: the 1936 volume, printed page 614, OCR page index 832, block index 11. The OCR page index and printed page number differ.
For a similar discussion with your coding agent, start with:
Read this OCR sample. What could Python extract reliably using explicit rules? Show the clauses those rules would struggle with. Propose fields and two example records, then explain how Python and a model could share the work.
Why regular expressions alone were insufficient
This is a familiar problem for historians: semi-structured data. Names, offices, places and dates recur, but their relationships are expressed in varied prose. The irregularity belongs to the source as well as to the OCR.
| Example | What a reading must establish |
|---|---|
| Balmer: “senr. dep. P.M.G., do., 1935” | The new position inherits the preceding place |
| Blakely: “B. Honduras, Apl., 1898” | “B.” abbreviates British Honduras; April 1898 dates the appointment |
| Trimen: lecturer, 1867–1875; British Museum assistant, 1869–1879 | Two appointments overlap; one does not end when the other begins |
A regular expression can recognise a year or a specified abbreviation. Binding that year to the correct posting, carrying a place across an omitted phrase, or separating concurrent jobs needs more context. A fixed grammar could handle the regular cases, but adding exceptions did not make it a reliable parser for the whole collection.
What the rules did at scale
The project’s real rule-based parser is 258 lines of Python. It splits a career at semicolons, knows months, abbreviations and ditto marks, and refuses an entry whole when a dated clause will not parse. Run on every printed biography in 66 editions of the Colonial Office List, it parsed 157,108 of 202,100 (78%); its share rises from 61% in the 1888 edition to 96% in 1960 as the house style became more regular. Entries it refused, such as Wodehouse’s prose entry of 1867, were read by Qwen.
The harder problem is what the rules accept. Balmer’s “senr. dep. P.M.G., do., 1935” came out as the Dominions Office, because the abbreviation “do” fired before the ditto rule. Connolly’s “agt. gen. in London for Malta, 1930” was accepted with Western Australia carried forward. A later pass recovered 3,755 places the rules had dropped because the colony sat mid-clause. Each new rule can collide with another, and refusal only catches clauses that fail to parse, not clauses parsed wrongly. The parser’s saved output and per-edition counts are extracted with scripts/extract_rules_examples.py from the Imperial Careers repository.
Python remained useful for splitting the input, parsing regular clauses, managing the run and checking outputs. We developed a hybrid so a model could read the difficult clauses within an agreed structure.
Develop and check the hybrid
The collaboration with Claude Code produced the schema, Python stages and extraction prompt. We settled the fields on a sample, inspected example records against their source, and revised the rules. The in-session runbook records batches of reference examples followed by Python validation. The model inside the eventual batch script was Qwen; Claude Code was the environment used to develop the machinery.
The resulting workflow had two stages.
Stage 1: reading every edition. To link the same person across editions, every printed biography was read. The rule-based parser handled the regular entries and refused the rest whole; the refused entries went to a model. For the Colonial Office List the first residue went to a batch API with a cheap commercial model; later residue went to Qwen. The entries were then linked into people (step 4).
Stage 2: the final records. For the 65,239 deduplicated careers (34,743 Colonial Office in June 2026, 30,496 India Office a week later), the model read every career, because the rules’ accepted output could be silently wrong:
- Python first. A Python pass expands the abbreviations it recognises (“Feb.” to “February”, “acctnt.” to “accountant”) and leaves everything else exactly as printed.
- The model reads every career. Binding position to place to time span across run-on, elided, “ditto” and concurrent clauses. One entry in, one JSON object out, from an open model on a desktop GPU.
- Python again. A validator that checks every place and year the model emitted against the source text and drops what it cannot find there. Then a post-normalisation pass that applies the deterministic fixes found by the audit, logging every change.
The schema
One object per entry: surname, given_names, birth_year (only if printed), education, honours with award years, events, and terminal. One event per posting: position and place as printed, year_start, year_end, place_inherited, is_acting, and civil or military. The agent and the historian settled it on a sample before any batch ran.
The prompt, exactly as sent
The system message for the approximately 65,000 career records in the structuring runs (file):
You convert ONE printed Colonial Office List biographical entry into a JSON career record. Extract ONLY what is printed; never invent or infer facts.
Rules:
- events: one object per distinct posting. EXPLODE compound/concurrent clauses into separate events (e.g. 'reg.-gen., 1887, and M.L.C., 1889' = two events; 'ag. comsnr., 1915-16; and from 1919' = two events).
- position: as printed (the normalized expansion is fine), e.g. 'assistant colonial secretary'.
- place: the colony, territory, OR locality named in the clause, AS PRINTED (e.g. 'Ceylon', 'Negombo', 'N. Nigeria'). If the clause names no place, carry over the previous clause's place and set place_inherited=true; do NOT guess or upgrade a locality to its parent colony — record exactly what is printed. Inheritance stops when a transfer is stated.
- year_start/year_end: the posting's years; year_end null if a single year.
- is_acting: true when the post is acting ('ag.'/'acting').
- org_type: 'military' for army/navy/regiment/expedition/campaign postings, else 'civil'.
- honours: decorations/orders with their AWARD year if printed in parentheses (e.g. 'C.M.G. (1905)'); these award-years are NOT career events.
- birth_year: only if explicitly printed ('B. 1872'); else null. NEVER infer from the earliest career date.
- education: the 'ed. ...' string as one metadata string; university class honours stay here, NOT as events.
- terminal: {kind: retired|resigned|died|pensioned|other, year} if the entry states the career ended, else null.
Return ONE JSON object with EXACTLY these top-level keys: surname, given_names, birth_year, education, honours, events, terminal. Split the headword into surname + given_names (no 'name'/'person' key). Each honour is {"award": str, "year": int|null} (no 'decoration'/'order' key).
The user message is the entry alone, after the abbreviation pass. Trimen’s 1889 entry goes in like this (file):
TRIMEN, HENRY, M.B. (Lond.), F.R.S., F.L.S. —Formerly lecturer on botany at St. Mary's Hospital, London, 1867 to 1875; was senior assistant to department of botany, British Museum, 1869 to 1879; director Royal Botanical Gardens, Ceylon, February, 1880; author of numerous works and papers on pure and applied botany.
Three things to notice. Every rule in the list exists because an earlier run broke it; the rule about birth years is there because a model had been inferring them from the first appointment. The last paragraph pins the key names, because a bake-off found the model drifting to name and decoration without it. And nothing in the prompt is about the historical question; it is entirely about what the page says and how to hold it.
The overnight run
| Colonial Office List | India Office List | |
|---|---|---|
| Model | Qwen3.6-35B-A3B, served by vLLM on an NVIDIA DGX Spark desktop GPU | same |
| Settings | temperature 0, reasoning off, 8 to 16 workers, 4 retries | same |
| Entries | 34,743 | 30,496 |
| Rate once warm | about 1 a second | 1.03 a second |
| Elapsed | about a day including setup, 19–20 June | 8.24 hours, overnight 25–26 June |
| Failed to return valid JSON | 55, under 0.2 per cent |
The two structuring runs cover 65,239 career records: 34,743 Colonial Office and 30,496 India Office records, after preliminary chaining and deduplication. These are not every printed appearance of an entry, and not the final person counts after further identity resolution. The Colonial Office run instructions describe the deduplicated input; the India Office method distinguishes printed biographies, career chains, structuring and final merges.
The handoff note records the setting that matters most: reasoning off. With it on, long entries eat the token budget and return no JSON, a 67 per cent failure rate. Thinking off, temperature zero: the same finding every run on this corpus has made.
The model was chosen first through OpenRouter, a hosted service, by running several open models on the same 40 entries; the production run then used the same model on local hardware. The glossary compares the hosted, desktop and cluster routes, with a worked cost estimate at current hosted prices.
The India Office List needed one new piece of code, a Python segmenter for its different OCR format, written that morning. The prompt, the model, the validator and the post-normaliser were reused unchanged through one environment variable. The graph was published that afternoon.
Two kinds of model
Claude in Claude Code structured about 200 entries in-session as the reference set and wrote the pipeline with the historian’s direction. The open model inside the script read the corpus. Every record says which path produced it. The runbook for the in-session loop is saved.
What came back
Trimen’s entry, structured (saved):
| position | place | years | |
|---|---|---|---|
| lecturer on botany | London | 1867–1875 | |
| senior assistant to department of botany | British Museum | 1869–1879 | overlaps the first: two concurrent posts, two events |
| director Royal Botanical Gardens | Ceylon | 1880– |
Plus education: "M.B. (Lond.)", two honours without years, no birth year, no terminal event. In the published graph the first and third events are placed; “British Museum” is an institution, not a place, and stays unplaced.
The audit
Structured records were read against their source by a reviewer model, in three rounds, and the findings categorised (catalogue, round 1 pairs). The figures are before the post-normaliser described below:
| Round | Sample | Fully clean | Major error |
|---|---|---|---|
| 1 | every tenth of the first 1,859 records: 186 | 129 (69%) | 6 records (3.2%); 8 findings |
| 2 | 100 spread across the corpus | 51 (51%) | 9 (about 9%) |
| 3 | 300 spread across the corpus | about 149 (50%) | not tallied |
Round 1 sampled only the first 1,859 records; it was not spread across the corpus. Rounds 2 and 3 cover a broader range. These are sample results, not a measured accuracy rate for the finished graph. A model did the reading in all three; the examples below were checked by hand.
Three of the findings, source beside record (saved):
The ditto that was not resolved. Balmer’s last clause, “senr. dep. P.M.G., do., 1935”, came back with place: "do.". The rule says inherit; the model copied the abbreviation.
The appointment date that became a birth year. “BLAKELY, DONALD QUENTIN.—Clk., atty.-gen.’s off., B. Honduras, Apl., 1898.” came back with birth_year: 1898 and an event with no year. The prompt forbids exactly this. The same record expanded “off.” to “officer” instead of “office”.
The knighthood split in two. “KT. BACH. (1920)” came back as two honours, KT and BACH (1920). Three records in the sample did the same.
Why the fix was not a bigger model
The errors are not random. They are a handful of systematic patterns: a short list of abbreviations mis-expanded the same way every time, ditto marks left literal, one compound honour split, “mentioned in despatches” dropped. Each has a deterministic fix, and those fixes are the post-normalisation pass: merge KT + BACH into Knight Bachelor; forward-fill a “do.” place from the previous event and set the flag; strip honorifics folded into given names; “officer in chief” back to “officer in charge”; drop a military tag with no military signal in the position. Every change is appended to a list on the record, so nothing is silent. The “off.” expansion is fixed one step earlier, by not pre-expanding ambiguous abbreviations at all.
The fixes do not establish the accuracy of the finished corpus. That needs a new audit after normalisation, including records changed by each rule and a fresh sample. A more capable model would also need that audit.
One correction, checked
Here is a small replay you can run without a model or a network. It uses the three saved audit records above: twelve events in all. It changes only a literal “do.” or “ditto” when the preceding event has a resolved place.
| Balmer’s fourth event | |
|---|---|
| Source | “senr. dep. P.M.G., do., 1935” |
| Before | place: "do.", place_inherited: true |
| After | place: "Kenya-Uganda-Tanganyika", place_inherited: true |
| Reason kept with the change | Printed ditto; the previous event supplies the place |
Checked: one event changed; the other eleven events and all other fields stayed unchanged. A second run made no further changes. Constructed boundary cases with no preceding resolved place stayed unresolved. Blakely’s birth-year error and Connolly’s split knighthood remain: this rule fixes one class of error.
Saved before-and-after records and checks. To reproduce from the workshop repository:
python3 scripts/demo_ditto_fix.pyThis is a workshop replay on three records, not a rerun of the published graph or a post-fix accuracy estimate.
You decide: what is an event. Whether an acting appointment is a posting. Whether “called to the bar, 1879” belongs in the career. Whether “leader of leg. coun. from 1906 to 1911” is a position or a description. The prompt records those decisions; the audit checks whether the model kept them; the post-normaliser enforces the ones that can be enforced.