Skip to main content
Every object ingested into Mixpeek is fingerprinted with a SHA-256 content hash, and every document produced from it carries that hash in its system envelope alongside a full lineage record. Together they answer the two questions every pipeline eventually asks:
  1. Have I already processed this? — the content hash.
  2. What downstream work is stale now that this changed? — the lineage chain.

​
Content hashing

The hash is computed from the source content at ingestion and stored:
  • on the object (in the bucket), and
  • on every document derived from it (content_hash in the document’s system envelope).
Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale. “Latest version” of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.

​
Dedup strategy

When you submit a batch, dedup_strategy controls how objects that were already processed in a prior batch are handled. Dedup is scoped to the (bucket, collection) pair — an object is a duplicate if the target collection already has documents produced from that same source object.
skip reprocesses an object when its content or the extractor configuration changed since it was last extracted, and skips everything else. It does not delete documents a re-run no longer produces, for example the extra chunks left behind when a new chunking setting yields fewer of them. For a clean rebuild that removes those, use replace.

​
The re-processing cascade

Derived assets stay in sync with their source through the combination of both mechanisms:
  1. A source object’s content changes → its hash changes.
  2. Re-submitting with replace (or re-syncing the source) reprocesses it — and every derived document downstream in its lineage (transcode → embed → tag) is rebuilt from the new source.
  3. Sources whose hash didn’t change are skipped — unchanged content is never re-paid for.
This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document is the record, the hash is the version, and lineage is the dependency graph.