> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deduplicate

> Remove duplicate documents by field match or content similarity

<Frame>
  <img src="https://mintcdn.com/mixpeek/TwtTrae3Fi3EFJ72/assets/retrievers/deduplicate.svg?fit=max&auto=format&n=TwtTrae3Fi3EFJ72&q=85&s=38b32d27d3be3567bcb3599d6630cbff" alt="Deduplicate stage showing removal of duplicate documents" width="1000" height="380" data-path="assets/retrievers/deduplicate.svg" />
</Frame>

The Deduplicate stage removes duplicate documents from the result set based on exact field matching or content similarity. This is analogous to SQL's `DISTINCT`, MongoDB's `$group` with `$first`, and Elasticsearch's field collapsing.

<Note>
  **Stage Category**: REDUCE (Removes duplicates)

  **Transformation**: N documents → M documents (M ≤ N, duplicates removed)
</Note>

## When to Use

| Use Case | Description |
| - | - |
| **URL deduplication** | One result per source URL after web enrichment |
| **Author collapse** | Keep one result per author |
| **Content dedup** | Remove near-identical text chunks |
| **Multi-source merge** | Remove overlapping results from multiple searches |
| **Query expansion cleanup** | Remove duplicates from expanded query results |

## When NOT to Use

| Scenario | Recommended Alternative |
| - | - |
| Grouping with aggregation | `group_by` stage |
| Sampling unique categories | `sample` with stratified |
| Limiting result count | `limit` stage |
| Filtering by criteria | `attribute_filter` |

## Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `strategy` | string | `field` | Dedup method: `field` (exact match) or `content` (similarity) |
| `fields` | list\[string] | *required for field* | Field paths to compare for deduplication |
| `content_field` | string | `content` | Text field for content-based dedup |
| `similarity_threshold` | float | `0.95` | Similarity threshold for content dedup (0.0-1.0) |
| `keep` | string | `first` | Which duplicate to keep: `first` or `last` |
| `case_sensitive` | boolean | `true` | Whether string comparisons are case-sensitive |

## Deduplication Strategies

| Strategy | Performance | Best For |
| - | - | - |
| `field` | O(N) hash-based | Exact field matching (URL, ID, title) |
| `content` | O(N²) pairwise | Near-duplicate text detection |

## Configuration Examples

<CodeGroup>
  ```json Deduplicate by URL theme={null}
  {
    "stage_name": "deduplicate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "deduplicate",
      "parameters": {
        "strategy": "field",
        "fields": ["metadata.source_url"],
        "keep": "first"
      }
    }
  }
  ```

  ```json Case-Insensitive Author Dedup theme={null}
  {
    "stage_name": "deduplicate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "deduplicate",
      "parameters": {
        "strategy": "field",
        "fields": ["metadata.author"],
        "case_sensitive": false
      }
    }
  }
  ```

  ```json Multi-Field Dedup theme={null}
  {
    "stage_name": "deduplicate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "deduplicate",
      "parameters": {
        "strategy": "field",
        "fields": ["metadata.author", "metadata.title"]
      }
    }
  }
  ```

  ```json Content Similarity Dedup theme={null}
  {
    "stage_name": "deduplicate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "deduplicate",
      "parameters": {
        "strategy": "content",
        "content_field": "content",
        "similarity_threshold": 0.9,
        "keep": "first"
      }
    }
  }
  ```
</CodeGroup>

<Tip>
  For best results, place deduplicate after sorting/reranking so that `keep: "first"` retains the highest-scored duplicate. This ensures you keep the most relevant version of each document.
</Tip>

## Performance

| Metric | Value |
| - | - |
| **Latency** | \< 5ms (field) / 10-100ms (content) |
| **Memory** | O(N) hash set (field) / O(N) content cache (content) |
| **Cost** | Free |
| **Complexity** | O(N) field / O(N²) content |

## Common Pipeline Patterns

### Web Search Deduplication

```json theme={null}
[
  {
    "stage_name": "feature_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 50}],
        "final_top_k": 50
      }
    }
  },
  {
    "stage_name": "external_web_search",
    "stage_type": "apply",
    "config": {
      "stage_id": "external_web_search",
      "parameters": {
        "query": "{{INPUT.query}}",
        "num_results": 10
      }
    }
  },
  {
    "stage_name": "deduplicate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "deduplicate",
      "parameters": {
        "strategy": "field",
        "fields": ["metadata.source_url"]
      }
    }
  }
]
```

### Cross-Collection Dedup

```json theme={null}
[
  {
    "stage_name": "feature_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [{"feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": {"input_mode": "text", "value": "{{INPUT.query}}"}, "top_k": 100}],
        "final_top_k": 100
      }
    }
  },
  {
    "stage_name": "rerank",
    "stage_type": "sort",
    "config": {
      "stage_id": "rerank",
      "parameters": {
        "inference_name": "BAAI__bge_reranker_v2_m3",
        "query": "{{INPUT.query}}",
        "document_field": "content"
      }
    }
  },
  {
    "stage_name": "deduplicate",
    "stage_type": "reduce",
    "config": {
      "stage_id": "deduplicate",
      "parameters": {
        "strategy": "content",
        "content_field": "content",
        "similarity_threshold": 0.85
      }
    }
  }
]
```

## Error Handling

| Error | Behavior |
| - | - |
| Field doesn't exist | Documents with missing fields have `None` as key value |
| All unique documents | Returns all documents unchanged |
| Empty input | Returns empty result set |
| Single document | Returned as-is (no duplicates possible) |

## Related

* [Group By](/docs/retrieval/stages/group-by) - Group documents with aggregation
* [Limit](/docs/retrieval/stages/limit) - Truncate results after deduplication
* [Sample](/docs/retrieval/stages/sample) - Random sampling (different from dedup)
* [Unwind](/docs/retrieval/stages/unwind) - Inverse: expand grouped items


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.