Ingestion at Scale: Why the Obvious Loop Dies at a Million Files

for key in bucket:

doc = extract(key)

index(doc)

That's correct for a thousand files. At a million, a corrupt video or a 2GB PDF throws, and the loop dies with 400,000 files left unprocessed and no record of where it stopped.

A million files isn't a bigger loop. It's a different program, and five constraints separate them.

Batch. Split the corpus into units of 1,000 to 10,000 objects and track each as pending, in-flight, done or failed. Size them to finish in minutes rather than hours. That single choice gives you a progress ledger, a place to parallelise, and a retry granularity that isn't the whole run.

Bounded concurrency. A fixed worker pool caps memory, connections and spend. Unbounded parallelism doesn't go faster past the first bottleneck, it just fails in more places at once.

Backpressure. A lister enumerates a million keys in seconds while extraction handles a hundred a second. An unbounded queue between them grows until it runs out of memory. The queue has to be able to stall the producer.

Checkpointing. A run that dies at 612,000 of a million should come back at 612,000 and be able to name the three batches that failed.

Idempotent retries. Retries are certain at this scale. If a retry re-extracts a file that already succeeded, the cost is real money on GPU time, and the bill arrives without an error to explain it. That makes it the failure most likely to reach a finance dashboard before an engineering one.

The deeper shift is the failure model. At a thousand files a failure is an exception you read. At a million it's a rate you manage. Some percentage of any real corpus is corrupt, oversized, or in a format nobody planned for, and the pipeline has to survive that rather than be surprised by it.

mxp.co/d/ingestion-at-scale

[image: ingestion-at-scale.png]