How Do I Ingest Millions of Files into a Search Index? (Images, Video, Documents at Scale)

The Short Answer

Ingesting a hundred files is a loop. Ingesting a million is a *system*, and it fails in ways the loop never does: one bad file stalls the whole run, a mid-run crash makes you start over, a retry silently re-pays for GPU extraction you already did, and unbounded parallelism melts your workers or trips every downstream rate limit. Doing it well means five things: batch the work into resumable units, bound the concurrency so you don't overwhelm workers or APIs, apply backpressure so producers can't outrun consumers, checkpoint so a failure resumes from where it stopped instead of from zero, and make retries idempotent so re-processing a file reuses prior work rather than paying for it twice. Get those five right and scale becomes boring, which is the goal.

Why "just loop over the files" breaks at scale

A naive script (list files, for-each, extract, index) works until it doesn't, and at a million files it won't:

One poison file halts everything. A corrupt video or a 2GB PDF throws, and an unguarded loop dies with 400k files left unprocessed and no record of where it stopped.

A crash means starting over. No checkpoint, no resume: an OOM at hour six sends you back to file zero, re-paying for everything already done.

Unbounded parallelism is its own outage. Fan out a million embed calls at once and you OOM your GPU workers, exhaust connections, and trip rate limits on every API you touch. Uncapped concurrency is slower than bounded concurrency once you count the retries.

Retries double your bill. The expensive step in multimodal ingestion is *extraction*, running models over pixels and audio. If a retry re-extracts what you already computed, every failure and every re-run silently re-pays full GPU cost. This is the single most expensive mistake at scale, and it's invisible until the invoice arrives.

The Five Constraints

1. Batch: make the unit of work resumable

Split the corpus into batches (say 1k-10k objects each) and track each batch's state: pending, in-flight, done, failed. The batch, not the individual file, is your unit of retry and resume. Batches give you a progress ledger ("612k of 1M done, 3 batches failed") and a natural place to parallelize. Size them so a batch finishes in minutes, not hours: small enough that a failure is cheap to retry, large enough that per-batch overhead is amortized.

2. Bounded concurrency: cap in-flight work

Run N batches concurrently where N is tuned to your slowest resource (usually GPU workers or a downstream API's rate limit), not to your file count. A semaphore or worker pool of fixed size is the whole mechanism. The right N maximizes throughput without pushing any stage into failure; past that point, adding concurrency only adds retries. Measure the bottleneck stage and size to it.

3. Backpressure: let consumers slow producers

If your lister enumerates a million object keys in seconds but extraction handles 100/sec, an unbounded queue between them grows until it OOMs. Backpressure means the producer blocks when the queue is full: a bounded queue, a paused cursor, or credit-based flow control. The system self-regulates to the speed of its slowest stage instead of buffering the whole corpus in memory.

4. Checkpointing: resume, don't restart

Persist progress durably (which batches are done, which object keys are processed) so any failure resumes from the last checkpoint. The test: kill the job at a random moment and restart it, a checkpointed pipeline picks up near where it stopped; an un-checkpointed one starts over. At a million files, "start over" can mean days and thousands of dollars re-paid, so checkpointing is not optional at scale.

5. Idempotent retries: never re-pay for work already done

Retries are inevitable, so make re-processing a file a no-op for the expensive parts. Key each unit of work by a stable identity (a content hash or object key + version) so a retry can look up "already extracted this" and reuse the result instead of re-running the model. This is where deduplication earns its keep: dedup by content means a file that appears twice, or a batch resubmitted after a partial failure, is extracted once. Without it, retries and re-runs re-pay full extraction cost; with it, they don't.

The Cost Trap Nobody Warns You About

Say it plainly because it's the mistake that hurts: at scale, the expensive resource is GPU extraction, and the default behavior of most "just re-run it" workflows is to re-extract. A heal, a resubmit, a retry after a transient error: each can quietly re-pay the full extraction bill for files that were already processed, because the system has no idea the work was done before. The fix is idempotency plus a dedup strategy that reuses prior extraction. Before you run a million-file ingest, answer one question: *when this retries or resumes, does it re-extract or reuse?* If you can't answer it, you will find out from the invoice.

Scale Changes the Failure Model

At 100 files

At 1,000,000 files

A failure is visible and rare, you re-run by hand	Failures are constant background noise, the system must self-heal
Restart from zero is cheap	Restart from zero costs days and real money: you must checkpoint
One thread is fine	You need bounded concurrency and backpressure or you cause an outage
Retries are free	Retries re-pay GPU cost unless they're idempotent + deduped
You watch it finish	You need a progress ledger and per-batch observability

Same task, different engineering. The move from a script to a system happens somewhere in the tens of thousands of files: plan for it before you cross that line, not after.

How to Run It on Mixpeek

Mixpeek is built around this shape so you don't assemble it yourself:

Batches are first-class. You submit objects as a batch and each batch has a tracked lifecycle and id, so you get the progress ledger and per-batch retry for free rather than bolting state onto a script. See the syncs & ingestion docs.

Extraction is a decomposition DAG. Each object is decomposed by feature extractors (scenes, faces, OCR, transcripts, embeddings) as a pipeline, and the platform handles the bounded concurrency and backpressure across that DAG instead of you tuning a worker pool.

Resubmit resumes, it doesn't restart. A resubmitted batch picks up rather than being treated as a new run, the changelog tracks the fix that stopped resubmitted batches from being killed as stalled, so a partial failure is cheap to recover.

A dedup strategy controls the re-pay question. Deduplication decides whether re-submitting already-processed objects re-runs extraction or reuses the prior result: the exact lever that stops a retry or heal from silently re-paying GPU cost. Choose it deliberately before a large run.

Vectors land where you want them. Extracted embeddings go into managed indexing or your own MVS vector store on object storage, so ingestion and retrieval share one data model. For the reliability side of large ingests, see production ingestion reliability; rate economics are on pricing.

The point is not that Mixpeek is magic: it's that the five constraints above are load-bearing at scale, and a platform that already implements batching, bounded concurrency, resumable checkpoints, and content dedup means you configure them instead of rebuilding them under deadline.

FAQ

How do I ingest millions of images or videos without blowing my GPU budget? Make retries idempotent and turn on content deduplication so that re-processing an already-extracted file reuses the prior result instead of re-running the model. The expensive step at scale is extraction (running models over pixels and audio); the default of naive re-run workflows is to re-extract, which silently doubles cost on every retry, heal, or resume. Answer "does this re-extract or reuse on retry?" before you start.

What batch size should I use for large-scale ingestion? Size batches so each finishes in minutes, not hours: commonly 1k-10k objects depending on file size and extraction cost. Small enough that a failed batch is cheap to retry, large enough that per-batch overhead is amortized. Then run a fixed number of batches concurrently, tuned to your slowest resource (usually GPU workers or a downstream rate limit), not to the total file count.

How do I make a huge ingest resumable so a crash doesn't restart it? Checkpoint durably: persist which batches and object keys are done so a restart resumes from the last checkpoint instead of file zero. Track per-batch state (pending / in-flight / done / failed) as a progress ledger. The test is simple: kill the job mid-run and restart it; if it picks up near where it stopped, you're checkpointed; if it starts over, you're not, and at a million files that difference is days and real money.

Why is my re-run or heal so expensive? Almost always because it re-extracts. If the retry/resume path isn't idempotent and content-deduplicated, every re-run re-pays full GPU extraction for files that were already processed. Fix it with a stable per-object identity (content hash or key+version) and a dedup strategy that reuses prior extraction; verify the behavior on a small batch before committing to a full run.

What if the extraction step is my own model? The scaling constraints above are unchanged, but versioning gets sharper: once your extractor has written vectors, its version is part of your data. See running your own model inside a managed search pipeline for the packaging contract and the ingest-versus-query-time trap that costs a full re-index when you get it wrong.