1.  [](/)
2.  [Code examples](/recipes)
3.  Multimodal Hybrid Search Pipeline

Back to code examples

Similar

# Multimodal Hybrid Search Pipeline

Combine vector search with keyword search (BM25) across text, images, and video for the most comprehensive multimodal retrieval system.

text

image

video

Production

[Run in Builder](/build?recipe=multimodal-hybrid-search)

Python

cURL

from mixpeek import Mixpeek

client \= Mixpeek(api\_key\="YOUR\_API\_KEY", namespace\="multimodal-search")

\# 1. Two collections, one extractor each: video and images through multimodal\_extractor, documents through text\_extractor

media \= client.buckets.create(

    bucket\_name\="media",

    bucket\_schema\={

        "properties": {

            "asset": {

                "type": "video",

            },

        },

    },

)

docs \= client.buckets.create(

    bucket\_name\="documents",

    bucket\_schema\={

        "properties": {

            "body": {

                "type": "text",

            },

        },

    },

)

media\_collection \= client.collections.create(

    collection\_name\="media",

    source\={"type": "bucket", "bucket\_ids": \[media\["bucket\_id"\]\]},

    feature\_extractor\={

        "feature\_extractor\_name": "multimodal\_extractor",

        "version": "v1",

    },

)

text\_collection \= client.collections.create(

    collection\_name\="documents",

    source\={"type": "bucket", "bucket\_ids": \[docs\["bucket\_id"\]\]},

    feature\_extractor\={

        "feature\_extractor\_name": "text\_extractor",

        "version": "v1",

        "parameters": {

            "split\_by": "paragraphs",

            "chunk\_size": 3,

        },

    },

)

\# 2. Upload and process

client.buckets.upload(

    media\["bucket\_id"\],

    blobs\=\[{"property": "asset", "type": "video", "data": "s3://your-bucket/media/q3-review.mp4"}\],

)

client.buckets.upload(

    docs\["bucket\_id"\],

    blobs\=\[{"property": "body", "type": "text", "data": "s3://your-bucket/docs/q3-report.md"}\],

)

client.collections.trigger(media\_collection\["collection\_id"\])

client.collections.trigger(text\_collection\["collection\_id"\])

\# 3. Dense search over both collections plus BM25 over document text (it reads the namespace text payload indexes), fused with RRF, then a rerank

retriever \= client.retrievers.create(

    retriever\_name\="hybrid-multimodal",

    collection\_identifiers\=\["media", "documents"\],

    input\_schema\={

        "query": {

            "type": "text",

            "required": True,

        },

    },

    stages\=\[

        {

            "stage\_name": "search",

            "stage\_id": "feature\_search",

            "parameters": {

                "searches": \[

                    {

                        "feature\_uri": "mixpeek://multimodal\_extractor@v1/vertex\_multimodal\_embedding",

                        "query": {

                            "input\_mode": "text",

                            "value": "{{INPUT.query}}",

                        },

                        "top\_k": 100,

                    },

                    {

                        "feature\_uri": "mixpeek://text\_extractor@v1/multilingual\_e5\_large\_instruct\_v1",

                        "query": {

                            "input\_mode": "text",

                            "value": "{{INPUT.query}}",

                        },

                        "top\_k": 100,

                    },

                    {

                        "feature\_uri": "mixpeek://text\_extractor@v1/multilingual\_e5\_large\_instruct\_v1",

                        "query": {

                            "input\_mode": "text",

                            "value": "{{INPUT.query}}",

                        },

                        "top\_k": 100,

                        "lexical": True,

                    },

                \],

                "fusion": "rrf",

                "final\_top\_k": 100,

            },

        },

        {

            "stage\_name": "rerank",

            "stage\_id": "rerank",

            "parameters": {

                "inference\_name": "BAAI\_\_bge\_reranker\_v2\_m3",

                "query": "{{INPUT.query}}",

                "document\_field": "text",

                "top\_k": 20,

            },

        },

    \],

)

\# 4. Search

results \= client.retrievers.execute(

    retriever\["retriever\_id"\],

    inputs\={

    "query": "quarterly revenue growth chart",

},

)

for doc in results\["documents"\]:

    print(doc\["document\_id"\], doc\["score"\])

### Feature Extractors

Multimodal Extractor

Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.

Text Embedding

Extract semantic embeddings from documents, transcripts and text content

### Retriever Stages

feature search

Search and filter documents by vector similarity using feature embeddings

filter

rerank

Rerank documents using cross-encoder models for accurate relevance

sort

## Frequently asked questions

### What is hybrid search?

Hybrid search combines dense vector similarity (semantic meaning) with sparse keyword matching (BM25) to capture both conceptual and exact-match relevance.

### How do I tune the vector vs BM25 weights?

Start with 0.7 vector / 0.3 BM25 for most use cases. Increase BM25 weight for domains with specific terminology (legal, medical).

### Does this work across languages?

Yes. The multilingual embedding models support 100+ languages for both queries and documents.

## Related code examples and resources

Pages on the same stages, extractors and patterns.

[

Extractor

### Text Embedding

Extract semantic embeddings from documents, transcripts and text content

Learn more





](/extractors/text)[

Extractor

### Multimodal Extractor

Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.

Learn more





](/extractors/multimodal)[

Recipe

### Multimodal Search with MVS

Build multimodal search by embedding different content types (text, images, video frames) with your own models and searching across them in a single MVS namespace. Use CLIP or any multimodal embedding model for cross-modal retrieval.

Learn more





](/recipes/mvs-multimodal-search)[

Recipe

### Document Intelligence Search

Extract and search through PDFs, presentations, and documents. Combines OCR, layout analysis, and semantic search for comprehensive document retrieval.

Learn more





](/recipes/document-search-pipeline)[

Recipe

### BYO Embeddings Vector Search

Bring pre-computed embeddings from any provider (OpenAI, Cohere, Together, etc.) and upsert them directly into MVS for instant vector search. No feature extractors, no pipelines -- just embeddings in, results out.

Learn more





](/recipes/mvs-byo-embeddings-search)[

Recipe

### Multimodal RAG Pipeline

Build a retrieval-augmented generation system that works with text, images, and video. Feed relevant multimodal context to LLMs for grounded responses.

Learn more





](/recipes/multimodal-rag-pipeline)