Your own data

Minutes 56–60. A practical next step for another collection.

  • fingerprint: a short code computed from the data, stored with each decision so it cannot be attached to a different version.
  • precision and recall: of what you found, how much was right; of what was there, how much you found.
  • stratified sample: a sample built slice by slice so the hard cases are represented.
  • frontier model: a model at the leading edge of capability; used here in a coding agent to write and repair the pipeline.

All terms: glossary.

The published session description invited you to bring messy historical datasets. An hour is not enough to work on them together, but the sequence transfers. Here is how to start, in the order that worked for Imperial Careers.

Before you open a model

  1. Pick one question and one small sample. Twenty records you can check by hand beat two thousand you cannot. Include at least one hard case.
  2. Keep the source reachable. Every record needs a reference that gets a reader back to the page: volume, page, line or offset. If your text came from OCR, keep the page images available. You will need them.
  3. Decide what a record is. A person? A posting? A mention? Imperial Careers has one person per resolved biography and one event per posting, and the difference between “entry instances” and “people” is the whole of step 4.
  4. Decide what the output looks like. A map, a timeline, a network. Build it early, from the first sample. It is your error detector.

The sequence

Step Ask the model to You
Structure Read the sample and propose fields, with two example records; write the prompt rules down Decide what the question requires and what the source can support
Extract Process the sample inside a script, keep printed wording, put inferences in separate fields, report failures Audit every tenth record against the source; sort the findings into patterns
Clean Cluster candidate duplicates and argue each case from a dossier Set the merge rules; decide the hard cases; keep the reasons
Ground Retrieve identities with a verifiable lookup, save the evidence and the date, gate on type and dates Judge the links that matter; set the gate for the rest
Visualise Build the view the question needs Look for what is in the wrong place
Fix Find other records with the same cause, change the rule, rerun, show what changed Check the fix and the previously accepted cases that moved

Rules that saved the project trouble

  • Local identifiers first. Every record gets a stable local id before any external one. Add a Wikidata link only after retrieval and review support it. Nothing suitable? Keep the local id.
  • No identifiers from memory. A model will offer plausible Q-numbers. Treat any identifier that was not retrieved by a tool as unverified.
  • Ground to the historical entity, not the modern one. The Colony of Victoria, not the state; British Raj, not India. Where a surface means different places for different people, resolve per person, not per string.
  • Ties resolve to no merge. A missed link is recoverable. A false merge can silently corrupt later analysis.
  • Model decisions and human decisions in different columns. Never overwrite one with the other. Export human decisions with a fingerprint of the data they were made on.
  • Fix the rule, not the record. Then look at everyone the rule touched.
  • Make the correction a test. The probe that found the defect goes into the test suite.
  • Report what the sample can support. Handpicked cases explain failure modes. A stratified sample measures precision for one stratum. Recall needs labelled true matches, including ones the system missed. An oversampled stratum is not the population; report its result separately or weight it appropriately.

Tools, and what they cost at scale

Two different jobs need two different kinds of model.

  • The agent that writes the pipeline reads your sample, proposes the structure, writes and tests the scripts, builds the review views and the atlas. This is hours of a frontier model’s time per project, in a coding agent. Imperial Careers used Claude in Claude Code.
  • The model inside the pipeline reads every entry and judges every cluster. At corpus scale, measure cost and throughput on a sample before choosing a model. Imperial Careers used Qwen3.6-35B-A3B on a DGX Spark desktop GPU for structuring, at about one entry a second, and Qwen3.8-27B on Alliance clusters for the never-biographied matching, with the frontier model’s in-session work as the reference set to measure it against.

The agent can write the pipeline so that the cheap model does the reading and the agent is only called back when something fails. Which operation produced a given record should remain visible in the output.

For your first trial, you do not need a university GPU. Use twenty records you can check, a hosted model or your existing coding agent, and a small output table. Record input and output tokens, charges, elapsed time, failures and review time. University computing also has a cost even when there is no bill per request. For scale: reading all 30,496 India Office entries through a hosted Qwen model would cost roughly $22–45 at October 2026 prices (worked estimate).

Take-home instruction for your agent:

Read these twenty raw entries. Explain which parts Python rules can handle reliably and which clauses need interpretation. Propose a hybrid approach, a schema and two example records before processing the rest. Keep a local record id and a source reference. Preserve the wording; put interpretations in separate fields. Show omissions, unresolved cases and validation failures. After my review, write a script that saves each result and can resume without repeating completed records.

The presenter’s environment uses Claude Code as the coding agent and the WikidataMCP server for lookups. Any coding agent that can run code, read files and call a lookup tool will do. See Run it yourself.