1.  [](/)
2.  [Code examples](/recipes)
3.  Multimodal RAG Pipeline

Back to code examples

Enhanced

# Multimodal RAG Pipeline

Build a retrieval-augmented generation system that works with text, images, and video. Feed relevant multimodal context to LLMs for grounded responses.

text

image

video

Multi-Tier

[Run in Builder](/build?recipe=multimodal-rag-pipeline)

Python

cURL

from mixpeek import Mixpeek

client \= Mixpeek(api\_key\="YOUR\_API\_KEY", namespace\="rag-kb")

\# 1. A bucket for the knowledge base, and a collection that splits every document

\#    into paragraphs and embeds each one

bucket \= client.buckets.create(

    bucket\_name\="knowledge-base",

    bucket\_schema\={"properties": {"document": {"type": "text"}}},

)

collection \= client.collections.create(

    collection\_name\="knowledge-base",

    source\={"type": "bucket", "bucket\_ids": \[bucket\["bucket\_id"\]\]},

    feature\_extractor\={

        "feature\_extractor\_name": "text\_extractor",

        "version": "v1",

        "parameters": {

            "split\_by": "paragraphs",

        },

    },

)

\# 2. Upload and process

client.buckets.upload(

    bucket\["bucket\_id"\],

    blobs\=\[{"property": "document", "type": "text", "data": "s3://your-bucket/knowledge-base/board-minutes-2026-08.md"}\],

)

client.collections.trigger(collection\["collection\_id"\])

\# 3. The retriever finds passages, reranks them and writes the answer. BM25 reads the

\#    namespace's text payload indexes, so declare one on text before relying on it.

retriever \= client.retrievers.create(

    retriever\_name\="rag-kb",

    collection\_identifiers\=\["knowledge-base"\],

    input\_schema\={"query": {"type": "text", "required": True}},

    stages\=\[

        {

            "stage\_name": "search",

            "stage\_id": "feature\_search",

            "parameters": {

                "searches": \[

                    {

                        "feature\_uri": "mixpeek://text\_extractor@v1/multilingual\_e5\_large\_instruct\_v1",

                        "query": {"input\_mode": "text", "value": "{{INPUT.query}}"},

                        "top\_k": 50,

                    },

                    {

                        "feature\_uri": "mixpeek://text\_extractor@v1/multilingual\_e5\_large\_instruct\_v1",

                        "query": {"input\_mode": "text", "value": "{{INPUT.query}}"},

                        "top\_k": 50,

                        "lexical": True,

                    },

                \],

                "fusion": "rrf",

                "final\_top\_k": 30,

            },

        },

        {

            "stage\_name": "rerank",

            "stage\_id": "rerank",

            "parameters": {

                "inference\_name": "BAAI\_\_bge\_reranker\_v2\_m3",

                "query": "{{INPUT.query}}",

                "document\_field": "text",

                "top\_k": 8,

            },

        },

        {

            "stage\_name": "answer",

            "stage\_id": "summarize",

            "parameters": {

                "prompt": "Answer the question {{INPUT.query}} using only these numbered passages, and cite the passage numbers. {{DOCUMENTS}}",

                "provider": "google",

                "model\_name": "gemini-2.5-flash-lite",

                "content\_field": "text",

                "output\_field": "answer",

                "include\_sources": True,

            },

        },

    \],

)

results \= client.retrievers.execute(retriever\["retriever\_id"\], inputs\={"query": "What were the key decisions from the last board meeting?"})

answer \= results\["documents"\]\[0\]

print(answer\["answer"\])

\# The summary document lists the documents it read; fetch them for citations

for i, document\_id in enumerate(answer.get("source\_document\_ids") or \[\], 1):

    source \= client.documents.get(collection\["collection\_id"\], document\_id)

    print(f"\[{i}\]", source.get("root\_object\_id"), source.get("text"))

### Feature Extractors

Text Embedding

Extract semantic embeddings from documents, transcripts and text content

### Retriever Stages

feature search

Search and filter documents by vector similarity using feature embeddings

filter

rerank

Rerank documents using cross-encoder models for accurate relevance

sort

summarize

Condense multiple documents into a summary using an LLM

reduce

## Frequently asked questions

### Which LLMs work with this template?

Any LLM that accepts text context works: OpenAI GPT-4, Anthropic Claude, Google Gemini, or self-hosted models.

### How do I handle image context in RAG?

Mixpeek returns image descriptions and captions alongside retrieved images. Pass these text descriptions as context to your LLM.

### What chunk sizes work best for RAG?

Semantic chunking (default) automatically determines optimal chunk boundaries. For manual control, 512-1024 tokens per chunk is a good starting point.

## Related code examples and resources

Pages on the same stages, extractors and patterns.

[

Extractor

### Text Embedding

Extract semantic embeddings from documents, transcripts and text content

Learn more





](/extractors/text)[

Recipe

### Taxonomy Enrichment Pipeline

Automatically classify and tag content using custom taxonomies. Map your content to IAB categories, custom hierarchies, or industry-specific classifications.

Learn more





](/recipes/taxonomy-enrichment-pipeline)[

Recipe

### Content Clustering Pipeline

Automatically group similar content together using embedding-based clustering. Discover themes, identify duplicates, and organize large content libraries.

Learn more





](/recipes/clustering-pipeline)[

Recipe

### Metadata Enrichment Pipeline

Automatically enrich your data with extracted metadata: entities, topics, sentiment, language, and custom attributes. Transform raw content into structured, queryable data.

Learn more





](/recipes/metadata-enrichment-pipeline)[

Recipe

### Multimodal Hybrid Search Pipeline

Combine vector search with keyword search (BM25) across text, images, and video for the most comprehensive multimodal retrieval system.

Learn more





](/recipes/multimodal-hybrid-search)[

Recipe

### Clinical Documentation Structuring

Production-grade pipeline for ingesting clinical documents, scanned charts, EHR exports, wound photos, and therapy notes, and structuring them into coded fields aligned with MDS 3.0, PDPM, and CMS audit requirements. Combines OCR, clinical NER, taxonomy classification, and hybrid retrieval to turn unstructured bedside documentation into queryable, auditable data.

Learn more





](/recipes/clinical-documentation-structuring)