What it does
Historical state records are scanned images of typewritten and handwritten pages, frequently unindexed, often the only copy. The pipeline OCRs them, chunks them with their page coordinates intact, embeds them locally, and serves retrieval that cites back to the specific page image.
The corpus currently stands at 5,435 rare Texas historical documents and 1,065,357 indexed passages. Embedding and inference run locally through Ollama, which for an archive of this kind is a preservation decision as much as a cost one.
In the code
@dataclass(frozen=True)
class Passage:
doc_id: str # stable id of the source document
page: int # 1-indexed page in the original scan
bbox: tuple # coordinates on that page
text: str # OCR output for this span
ocr_conf: float # confidence, carried through to the citationIllustrative shape. The bounding box is what lets the interface highlight the exact region of the scan behind a sentence, and the OCR confidence travels with the passage so a low-confidence read is visible rather than laundered into clean prose.
How this differs from the ordinary version
The citation is an image, not a footnote
A retrieval system over historical material has to assume its own OCR is sometimes wrong. Resolving every claim to a highlighted region of the original scan lets a researcher check the machine rather than trust it.
Local-first because the corpus is irreplaceable
Uploading a 500 GB archive of one-of-a-kind records to a third-party API creates a dependency and a disclosure the material does not warrant. The whole pipeline runs on hardware the archive controls.
In the field
Why this is a border-region project
El Paso County records, water rights, land grants and rail history sit in exactly this kind of unindexed scanned material, and much of it is bilingual across a century of changing conventions. A retrieval layer that handles both languages and shows its sources is the difference between an archive that is stored and one that is usable.
Questions
- How large is the corpus?
- 5,435 documents and 1,065,357 indexed passages, drawn from roughly 500 GB of source scans.