Skip to main content
Every object ingested into Mixpeek is fingerprinted with a SHA-256 content hash, and every document produced from it carries that hash in its system envelope alongside a full lineage record. Together they answer the two questions every pipeline eventually asks:
  1. Have I already processed this? — the content hash.
  2. What downstream work is stale now that this changed? — the lineage chain.

​
Content hashing

The hash is computed from the source content at ingestion and stored:
  • on the object (in the bucket), and
  • on every document derived from it (content_hash in the document’s system envelope).
Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale. “Latest version” of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.

​
Dedup strategy

When you submit a batch, dedup_strategy controls how objects that were already processed in a prior batch are handled. Dedup is scoped to the (bucket, collection) pair — an object is a duplicate if the target collection already has documents produced from that same source object.
skip compares against prior processing state, not against your current extractor configuration. If you have changed extractor settings (different embedding model, new transcode profile) and want existing documents rebuilt under the new configuration, use replace — skip will happily keep serving documents produced under the old configuration, and re-running a batch with skip costs nothing precisely because it does nothing.

​
The re-processing cascade

Derived assets stay in sync with their source through the combination of both mechanisms:
  1. A source object’s content changes → its hash changes.
  2. Re-submitting with replace (or re-syncing the source) reprocesses it — and every derived document downstream in its lineage (transcode → embed → tag) is rebuilt from the new source.
  3. Sources whose hash didn’t change are skipped — unchanged content is never re-paid for.
This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document is the record, the hash is the version, and lineage is the dependency graph.