> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deduplication & Re-processing

> Content-hash dedup decides what work is skipped, replaced, or forced when objects are processed into collections — and lineage decides what re-runs when a source changes

Every object ingested into Mixpeek is fingerprinted with a **SHA-256 content hash**, and every document produced from it carries that hash in its system envelope alongside a full [lineage record](/docs/retrieval/lineage-traversal). Together they answer the two questions every pipeline eventually asks:

1. **Have I already processed this?** — the content hash.
2. **What downstream work is stale now that this changed?** — the lineage chain.

## Content hashing

The hash is computed from the source content at ingestion and stored:

* on the **object** (in the bucket), and
* on every **document** derived from it (`content_hash` in the document's system envelope).

Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale.

"Latest version" of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.

## Dedup strategy

When you submit a batch, `dedup_strategy` controls how objects that were **already processed in a prior batch** are handled. Dedup is scoped to the *(bucket, collection)* pair — an object is a duplicate if the target collection already has documents produced from that same source object.

| Strategy           | Behavior                                                           | Cost profile                                                                             |
| ------------------ | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| `skip` *(default)* | Objects that already have documents are not reprocessed            | Free for unchanged content — no extraction, no inference                                 |
| `replace`          | Existing documents are deleted and the object is fully reprocessed | Full extraction cost; use when the source content or the extractor configuration changed |
| `force`            | Process regardless, allowing duplicate documents                   | Full cost, duplicates allowed — rarely what you want outside testing                     |

```json theme={null}
{
  "bucket_id": "bkt_abc123",
  "collection_id": "col_xyz789",
  "dedup_strategy": "skip"
}
```

<Warning>
  `skip` compares against *prior processing state*, not against your current extractor configuration. If you have changed extractor settings (different embedding model, new transcode profile) and want existing documents rebuilt under the new configuration, use `replace` — `skip` will happily keep serving documents produced under the old configuration, and re-running a batch with `skip` costs nothing precisely because it does nothing.
</Warning>

## The re-processing cascade

Derived assets stay in sync with their source through the combination of both mechanisms:

1. A source object's content changes → its hash changes.
2. Re-submitting with `replace` (or re-syncing the source) reprocesses it — and every derived document downstream in its [lineage](/docs/retrieval/lineage-traversal) (transcode → embed → tag) is rebuilt from the new source.
3. Sources whose hash **didn't** change are skipped — unchanged content is never re-paid for.

This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document *is* the record, the hash *is* the version, and lineage *is* the dependency graph.

## Related

* [Lineage traversal](/docs/retrieval/lineage-traversal) — querying and walking derivation chains
* [Processing pipeline](/docs/platform/processing) — how objects become documents
* [Batch ingestion at scale](/docs/operations/batch-ingestion-at-scale) — batch mechanics, tiers, and monitoring