The Short Answer
Ingesting a hundred files is a loop. Ingesting a million is a *system*, and it fails in ways the loop never does: one bad file stalls the whole run, a mid-run crash makes you start over, a retry silently re-pays for GPU extraction you already did, and unbounded parallelism melts your workers or trips every downstream rate limit. Doing it well means five things: batch the work into resumable units, bound the concurrency so you don't overwhelm workers or APIs, apply backpressure so producers can't outrun consumers, checkpoint so a failure resumes from where it stopped instead of from zero, and make retries idempotent so re-processing a file reuses prior work rather than paying for it twice. Get those five right and scale becomes boring — which is the goal.
Why "just loop over the files" breaks at scale
A naive script — list files, for-each, extract, index — works until it doesn't, and at a million files it won't:
The Five Constraints
1. Batch — make the unit of work resumable
Split the corpus into batches (say 1k–10k objects each) and track each batch's state: pending, in-flight, done, failed. The batch — not the individual file — is your unit of retry and resume. Batches give you a progress ledger ("612k of 1M done, 3 batches failed") and a natural place to parallelize. Size them so a batch finishes in minutes, not hours: small enough that a failure is cheap to retry, large enough that per-batch overhead is amortized.
2. Bounded concurrency — cap in-flight work
Run N batches concurrently where N is tuned to your slowest resource (usually GPU workers or a downstream API's rate limit), not to your file count. A semaphore or worker pool of fixed size is the whole mechanism. The right N maximizes throughput without pushing any stage into failure; past that point, adding concurrency only adds retries. Measure the bottleneck stage and size to it.
3. Backpressure — let consumers slow producers
If your lister enumerates a million object keys in seconds but extraction handles 100/sec, an unbounded queue between them grows until it OOMs. Backpressure means the producer blocks when the queue is full — a bounded queue, a paused cursor, or credit-based flow control. The system self-regulates to the speed of its slowest stage instead of buffering the whole corpus in memory.
4. Checkpointing — resume, don't restart
Persist progress durably (which batches are done, which object keys are processed) so any failure resumes from the last checkpoint. The test: kill the job at a random moment and restart it — a checkpointed pipeline picks up near where it stopped; an un-checkpointed one starts over. At a million files, "start over" can mean days and thousands of dollars re-paid, so checkpointing is not optional at scale.
5. Idempotent retries — never re-pay for work already done
Retries are inevitable, so make re-processing a file a no-op for the expensive parts. Key each unit of work by a stable identity (a content hash or object key + version) so a retry can look up "already extracted this" and reuse the result instead of re-running the model. This is where deduplication earns its keep: dedup by content means a file that appears twice — or a batch resubmitted after a partial failure — is extracted once. Without it, retries and re-runs re-pay full extraction cost; with it, they don't.
The Cost Trap Nobody Warns You About
Say it plainly because it's the mistake that hurts: at scale, the expensive resource is GPU extraction, and the default behavior of most "just re-run it" workflows is to re-extract. A heal, a resubmit, a retry after a transient error — each can quietly re-pay the full extraction bill for files that were already processed, because the system has no idea the work was done before. The fix is idempotency plus a dedup strategy that reuses prior extraction. Before you run a million-file ingest, answer one question: *when this retries or resumes, does it re-extract or reuse?* If you can't answer it, you will find out from the invoice.
Scale Changes the Failure Model
| At 100 files | At 1,000,000 files |
| A failure is visible and rare — you re-run by hand | Failures are constant background noise — the system must self-heal |
| Restart from zero is cheap | Restart from zero costs days and real money — you must checkpoint |
| One thread is fine | You need bounded concurrency and backpressure or you cause an outage |
| Retries are free | Retries re-pay GPU cost unless they're idempotent + deduped |
| You watch it finish | You need a progress ledger and per-batch observability |
How to Run It on Mixpeek
Mixpeek is built around this shape so you don't assemble it yourself:
The point is not that Mixpeek is magic — it's that the five constraints above are load-bearing at scale, and a platform that already implements batching, bounded concurrency, resumable checkpoints, and content dedup means you configure them instead of rebuilding them under deadline.
FAQ
How do I ingest millions of images or videos without blowing my GPU budget? Make retries idempotent and turn on content deduplication so that re-processing an already-extracted file reuses the prior result instead of re-running the model. The expensive step at scale is extraction (running models over pixels and audio); the default of naive re-run workflows is to re-extract, which silently doubles cost on every retry, heal, or resume. Answer "does this re-extract or reuse on retry?" before you start.
What batch size should I use for large-scale ingestion? Size batches so each finishes in minutes, not hours — commonly 1k–10k objects depending on file size and extraction cost. Small enough that a failed batch is cheap to retry, large enough that per-batch overhead is amortized. Then run a fixed number of batches concurrently, tuned to your slowest resource (usually GPU workers or a downstream rate limit), not to the total file count.
How do I make a huge ingest resumable so a crash doesn't restart it? Checkpoint durably: persist which batches and object keys are done so a restart resumes from the last checkpoint instead of file zero. Track per-batch state (pending / in-flight / done / failed) as a progress ledger. The test is simple — kill the job mid-run and restart it; if it picks up near where it stopped, you're checkpointed; if it starts over, you're not, and at a million files that difference is days and real money.
Why is my re-run or heal so expensive? Almost always because it re-extracts. If the retry/resume path isn't idempotent and content-deduplicated, every re-run re-pays full GPU extraction for files that were already processed. Fix it with a stable per-object identity (content hash or key+version) and a dedup strategy that reuses prior extraction; verify the behavior on a small batch before committing to a full run.