Retrieval Cookbook

Every recipe below is a complete, copy-paste retriever config. Create the retriever once, then execute it with runtime inputs.

Stage shape. Every stage is { "stage_name", "stage_type", "config": { "stage_id", "parameters" } }. stage_name is your label; stage_id is the implementation (feature_search, rerank, attribute_filter, …); stage_type is the category (filter, sort, reduce, group, enrich, apply). See Retrievers and Stages.

Point feature_uri at your collection’s index. Each recipe’s feature_search uses a feature_uri for a specific extractor (e.g. mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding). That string must match the collection you built — a video_search collection uses mixpeek://universal_extractor@v1/gemini-embedding-2, not the multimodal one. A mismatched feature_uri silently returns 0 results. Find yours with GET /v1/collections/{id} → vector_indexes[].feature_uri — see Find your feature_uri.

Create + execute (every recipe uses this)

# Create the retriever (body = one of the recipes below)
curl -sS -X POST "$MP_API_URL/v1/retrievers" \
  -H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" -d @recipe.json
# -> returns { "retriever_id": "ret_..." }

# Execute it with runtime inputs
curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute" \
  -H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{ "inputs": { "query": "people discussing electric vehicles" } }'

Multimodal RAG

Retrieve context for an LLM across a video’s visual content and transcript, fuse with RRF, rerank, and format into a single prompt-ready context block.

{
  "retriever_name": "multimodal-rag",
  "collection_identifiers": ["col_videos"],
  "input_schema": { "query": { "type": "text", "required": true } },
  "stages": [
    {
      "stage_name": "search",
      "stage_type": "filter",
      "config": {
        "stage_id": "feature_search",
        "parameters": {
          "fusion": "rrf",
          "final_top_k": 50,
          "searches": [
            { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
              "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 },
            { "feature_uri": "mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1",
              "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
          ]
        }
      }
    },
    { "stage_name": "rerank", "stage_type": "sort",
      "config": { "stage_id": "rerank", "parameters": { "inference_name": "BAAI__bge_reranker_v2_m3", "query": "{{INPUT.query}}", "top_k": 10 } } },
    { "stage_name": "prepare", "stage_type": "apply",
      "config": { "stage_id": "rag_prepare", "parameters": { "max_tokens": 8000, "output_mode": "single_context", "citation": { "style": "numbered" } } } }
  ]
}

Result: the 10 most relevant moments, reranked, formatted into a cited context block ready to paste into an LLM prompt.

Hybrid Search (dense + keyword/BM25)

Fuse semantic recall (dense vectors) with exact-keyword precision (BM25) under RRF — so brand names, SKUs, and prices like $9.99 still match. Requires a text payload index (see Text Indexes).

{
  "retriever_name": "hybrid-search",
  "collection_identifiers": ["col_docs"],
  "input_schema": { "query": { "type": "text", "required": true } },
  "stages": [
    {
      "stage_name": "hybrid",
      "stage_type": "filter",
      "config": {
        "stage_id": "feature_search",
        "parameters": {
          "fusion": "rrf",
          "final_top_k": 25,
          "searches": [
            { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
              "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 },
            { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
              "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "lexical": true, "top_k": 100 }
          ]
        }
      }
    }
  ]
}

Result: 25 results ranked by fused semantic + keyword relevance. Use rrf here — it ranks by position, immune to the cosine-vs-BM25 score-scale mismatch.

Search Across Languages (cross-lingual)

Mixpeek’s text and multimodal embeddings are multilingual (E5-Large / Gemini, 100+ languages) and every language lands in the same vector space. So a query in one language retrieves semantically matching content in any language — no translation step, no per-language index, no language parameter. Query in English, match Farsi, Mandarin, or Spanish transcripts and captions.

{
  "retriever_name": "cross-lingual-search",
  "collection_identifiers": ["col_content"],
  "input_schema": { "query": { "type": "text", "required": true } },
  "stages": [
    { "stage_name": "search", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 25,
        "searches": [ { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } }
  ]
}

Result: an English query like "discussion of sanctions" returns segments whose transcript or caption is in another language — ranked by meaning, not keywords. For audio/video collections, transcription is produced in the source language (universal / audio extractors; Whisper covers 99 languages) and the multilingual embedding is what makes it retrievable cross-lingually — just point feature_uri at that extractor’s transcript embedding, e.g. mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1.

Mixpeek retrieves across languages; it does not translate. There is no translation output field — matched text comes back in its original language. If you need it rendered in the reader’s language (e.g. original + English side by side), append an llm_enrich stage that translates each result into a new field:

{ "stage_name": "translate", "stage_type": "enrich",
  "config": { "stage_id": "llm_enrich", "parameters": {
    "provider": "openai",
    "model_name": "gpt-4o-mini",
    "prompt": "Translate the following to English, preserving names and numbers. Return only the translation:\n\n{{DOC.content}}",
    "output_field": "translation" } } }

Each result then carries both content (original) and translation (English) for side-by-side display. This is the recommended path — reach for a custom extractor only if you need translation baked in at ingest time.

Video Moment Localization

Search a video and collapse matching segments into a handful of seekable moments with start/end timestamps.

{
  "retriever_name": "video-moments",
  "collection_identifiers": ["col_videos"],
  "input_schema": { "query": { "type": "text", "required": true } },
  "stages": [
    { "stage_name": "search", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 200,
        "searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 } ] } } },
    { "stage_name": "moments", "stage_type": "reduce",
      "config": { "stage_id": "moment_group", "parameters": {
        "parent_field": "source_object_id",
        "time_field": "start_time",
        "merge_tolerance_ms": 2000,
        "max_moments_per_parent": 5,
        "score_strategy": "max",
        "output_mode": "annotated" } } }
  ]
}

Result: per video, up to 5 merged moments with timestamps you can seek to.

time_field must point at the field holding each segment’s timestamp. For a plain text query use the segment’s own start_time. query_chunks is only present when you search with a file input via query preprocessing.

Face Search (1:N identification)

Find every clip a person appears in by passing a reference face image as a content query against the face embedding.

{
  "retriever_name": "face-search",
  "collection_identifiers": ["col_videos"],
  "input_schema": { "reference_face_url": { "type": "text", "required": true } },
  "stages": [
    { "stage_name": "face", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 20,
        "searches": [ { "feature_uri": "mixpeek://face_identity_extractor@v1/insightface__arcface",
          "query": { "input_mode": "content", "value": "{{INPUT.reference_face_url}}" }, "top_k": 50 } ] } } }
  ]
}

Result: the 20 closest face matches. A cosine score above ~0.30 typically indicates the same person (see Face Identity). This recipe chases one reference face. To match every new video against a maintained roster of named identities (a watchlist you add to and refine over time), build a reference collection instead — see Bootstrap a Labeled Dataset.

Reverse Image Search + Dedup

Find visually similar items from other sources, deduplicated.

{
  "retriever_name": "reverse-image",
  "collection_identifiers": ["col_catalog"],
  "input_schema": {
    "image_url": { "type": "text", "required": true },
    "exclude_source": { "type": "text" }
  },
  "stages": [
    { "stage_name": "similar", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 100,
        "searches": [ { "feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
          "query": { "input_mode": "content", "value": "{{INPUT.image_url}}" }, "top_k": 100, "min_score": 0.7 } ] } } },
    { "stage_name": "exclude_self", "stage_type": "filter",
      "config": { "stage_id": "attribute_filter", "parameters": {
        "field": "metadata.source", "operator": "ne", "value": "{{INPUT.exclude_source}}" } } },
    { "stage_name": "dedupe", "stage_type": "reduce",
      "config": { "stage_id": "deduplicate", "parameters": {
        "strategy": "field", "fields": ["metadata.source_url"], "keep": "first" } } }
  ]
}

Result: unique visually-similar items, excluding the original source.

Search OCR Text from Scanned PDFs

Scanned and archival PDFs are OCR’d — and VLM-corrected for low-confidence blocks — by the document graph extractor, which embeds each text block. So OCR’d text is searchable exactly like any other text: no separate OCR index or step. Optionally keep only high-confidence blocks, or target a layout type (e.g. only table blocks).

{
  "retriever_name": "ocr-search",
  "collection_identifiers": ["col_scanned_docs"],
  "input_schema": { "query": { "type": "text", "required": true } },
  "stages": [
    { "stage_name": "search", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 25,
        "searches": [ { "feature_uri": "mixpeek://document_graph_extractor@v1/intfloat__multilingual_e5_large_instruct",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
    { "stage_name": "high_confidence", "stage_type": "filter",
      "config": { "stage_id": "attribute_filter", "parameters": {
        "field": "confidence_tag", "operator": "in", "value": ["A", "B"] } } }
  ]
}

Result: the 25 best-matching blocks (each with page_number, bbox, object_type), limited to high-confidence OCR. Drop the high_confidence stage to include fair/poor blocks, or filter object_type to table/form to target structured regions.

For text burned into video frames (not PDFs), enable run_ocr on the multimodal extractor — it populates an ocr_text payload field you can filter on with attribute_filter. See also feature_search → Lexical (BM25).

Search + Classify

Search, then attach an LLM/taxonomy label to each result in one pipeline.

{
  "retriever_name": "search-classify",
  "collection_identifiers": ["col_content"],
  "input_schema": { "query": { "type": "text", "required": true } },
  "stages": [
    { "stage_name": "search", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 25,
        "searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
    { "stage_name": "label", "stage_type": "enrich",
      "config": { "stage_id": "llm_enrich", "parameters": {
        "provider": "openai",
        "model_name": "gpt-4o-mini",
        "prompt": "Classify this result: {{DOC.content}}",
        "output_field": "label",
        "output_schema": { "category": "string", "confidence": "number" } } } }
  ]
}

Result: 25 results, each annotated with a structured label. For NSFW/safety classification use the classify stage; for taxonomy-backed labeling see Taxonomies.

Scan a batch of inputs

Any retriever can run many queries at once — ideal for moderating an upload queue or scanning a catalog:

curl -sS -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/execute/batch" \
  -H "Authorization: Bearer $MP_API_KEY" -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{ "queries": [ {"inputs": {"image_url": "https://…/a.jpg"}}, {"inputs": {"image_url": "https://…/b.jpg"}} ] }'

The retriever is planned once and reused across the batch (query optimization).

Composing recipes

Stages are independent — add, remove, or reorder them. Common extensions: append a rerank for precision, an attribute_filter for metadata scoping (the optimizer pushes it down for you), or a rag_prepare to format for an LLM.

​Multimodal RAG

​Hybrid Search (dense + keyword/BM25)

​Search Across Languages (cross-lingual)

​Video Moment Localization

​Face Search (1:N identification)

​Reverse Image Search + Dedup

​Search OCR Text from Scanned PDFs

​Search + Classify

​Scan a batch of inputs

​Composing recipes

Multimodal RAG

Hybrid Search (dense + keyword/BM25)

Search Across Languages (cross-lingual)

Video Moment Localization

Face Search (1:N identification)

Reverse Image Search + Dedup

Search OCR Text from Scanned PDFs

Search + Classify

Scan a batch of inputs

Composing recipes