The pass that runs before the pipeline
The usual failure with a large inherited corpus is that ingestion starts before anyone knows what is in it, and the pipeline spends a week discovering that half the images are duplicates and the filenames encode identity in three incompatible schemes.
IMAGO makes that a separate, read only phase. The manifest pass walks the root, hashes the head of each file, buckets by extension, samples content for text and magic bytes for binaries, and counts hits for each candidate filename pattern. It embeds nothing, parses nothing, and touches no index. The output is a manifest and a printed table.
Sampling instead of reading everything
Once a bucket passes a threshold the pass switches from reading every file to sampling a fixed number, and hashes only the first few kilobytes of each. That keeps the inventory of a very large tree bounded in time while still being enough to detect duplicates and identify format.
In the code
# How many files to sample per bucket once a bucket exceeds this many entries.
SAMPLE_THRESHOLD = 100
SAMPLE_COUNT = 10
HEAD_BYTES = 4096 # bytes hashed for the rolling sha256 prefix
# Patterns to detect in image filenames. Each pattern logs a hit count so the
# linker can later choose a strategy.
FILENAME_PATTERNS: dict[str, re.Pattern[str]] = {
"uuid": re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", re.I),
"genus_species": re.compile(r"^[A-Z][a-z]+_[a-z]+", re.I),
"binomial_space": re.compile(r"^[A-Z][a-z]+ [a-z]+", re.I),
"numeric_id": re.compile(r"^\d{4,}"),
"specimen_prefix": re.compile(r"^(specimen|sp|holo|para|type)[_\-]?\d+", re.I),
}This is the part most pipelines skip and then pay for. Rather than assuming a naming convention and writing a linker against it, the manifest pass counts how many files match each of five plausible conventions and reports the tally, so the linking strategy is chosen from evidence about this corpus instead of from a guess about corpora in general. The two sampling constants are the same instinct applied to cost: hashing four kilobytes of head is enough to catch a duplicate or a truncated file, and once a bucket is clearly homogeneous there is no information left in reading the eleventh thousand example of it.
How this differs from the ordinary version
Inventory is a phase, not a side effect
Making the read only pass a first class step means it can be run on a corpus nobody has permission to modify, re-run after a delivery, and diffed between runs. Ingestion pipelines that discover structure as they go cannot do any of those.
The linking strategy is chosen from counts
Five naming conventions are tested and tallied. Whichever one the corpus actually uses wins, and a corpus that uses two shows up as two counts rather than as a silent half-failure later.
In the field
Inherited archives
Institutions hand over drives, not databases. The first honest question about a delivered archive is what is in it, and answering that without writing to it is what makes the second question, what it is worth indexing, answerable.
Questions
- Does the manifest pass modify anything?
- No. It is read only by design, writes a single manifest file outside the corpus, and touches no index.
- How does it stay fast on a very large tree?
- It hashes only the first four kilobytes of each file and switches to sampling ten files per bucket once a bucket passes a hundred entries.