1.  [](/)
2.  [Code examples](/recipes)
3.  Multimodal Knowledge Base

Back to code examples

# Multimodal Knowledge Base

Consolidate documents, videos, images, and audio into a single searchable knowledge base with RAG capabilities. Supports natural language Q&A across all content types, with citations linking back to the exact source document, video timestamp, or image.

text

video

image

audio

Production

[Run in Builder](/build?recipe=multimodal-knowledge-base)

Python

cURL

from openai import OpenAI

from mixpeek import Mixpeek

client \= Mixpeek(api\_key\="YOUR\_API\_KEY", namespace\="knowledge-base")

openai \= OpenAI(api\_key\="YOUR\_OPENAI\_KEY")

\# 1. PDFs become layout blocks, and videos become transcribed passages

docs\_bucket \= client.buckets.create(

    bucket\_name\="knowledge-docs",

    bucket\_schema\={"properties": {"document": {"type": "pdf"}}},

)

docs \= client.collections.create(

    collection\_name\="knowledge-docs",

    source\={"type": "bucket", "bucket\_ids": \[docs\_bucket\["bucket\_id"\]\]},

    feature\_extractor\={

        "feature\_extractor\_name": "document\_graph\_extractor",

        "version": "v1",

    },

)

videos\_bucket \= client.buckets.create(

    bucket\_name\="knowledge-videos",

    bucket\_schema\={"properties": {"video": {"type": "video"}}},

)

videos \= client.collections.create(

    collection\_name\="knowledge-videos",

    source\={"type": "bucket", "bucket\_ids": \[videos\_bucket\["bucket\_id"\]\]},

    feature\_extractor\={

        "feature\_extractor\_name": "multimodal\_extractor",

        "version": "v1",

        "parameters": {

            "split\_method": "silence",

            "run\_transcription": True,

            "run\_transcription\_embedding": True,

        },

    },

)

client.buckets.upload(

    docs\_bucket\["bucket\_id"\],

    blobs\=\[{"property": "document", "type": "pdf", "data": "s3://your-bucket/handbook/remote-work-policy.pdf"}\],

)

client.buckets.upload(

    videos\_bucket\["bucket\_id"\],

    blobs\=\[{"property": "video", "type": "video", "data": "s3://your-bucket/all-hands/2026-08.mp4"}\],

)

client.collections.trigger(docs\["collection\_id"\])

client.collections.trigger(videos\["collection\_id"\])

\# 2. One stage searches both collections: each search reads the index its collection writes

retriever \= client.retrievers.create(

    retriever\_name\="knowledge-base",

    collection\_identifiers\=\["knowledge-docs", "knowledge-videos"\],

    input\_schema\={"query": {"type": "text", "required": True}},

    stages\=\[

        {

            "stage\_name": "search",

            "stage\_id": "feature\_search",

            "parameters": {

                "searches": \[

                    {

                        "feature\_uri": "mixpeek://document\_graph\_extractor@v1/intfloat\_\_multilingual\_e5\_large\_instruct",

                        "query": {"input\_mode": "text", "value": "{{INPUT.query}}"},

                        "top\_k": 30,

                    },

                    {

                        "feature\_uri": "mixpeek://multimodal\_extractor@v1/multilingual\_e5\_large\_instruct\_v1",

                        "query": {"input\_mode": "text", "value": "{{INPUT.query}}"},

                        "top\_k": 30,

                    },

                \],

                "fusion": "rrf",

                "final\_top\_k": 12,

            },

        },

    \],

)

question \= "What is our company policy on remote work?"

results \= client.retrievers.execute(retriever\["retriever\_id"\], inputs\={"query": question})

\# 3. PDF blocks keep their text in text\_raw and video passages in transcription, so the

\#    numbered context is assembled here with a citation for each kind

passages \= \[\]

for i, doc in enumerate(results\["documents"\], 1):

    if doc.get("text\_raw"):

        passages.append(f"\[{i}\] {doc\['text\_raw'\]} (PDF {doc.get('root\_object\_id')}, page {doc.get('page\_number')})")

    else:

        passages.append(f"\[{i}\] {doc.get('transcription')} (video {doc.get('root\_object\_id')} at {doc.get('start\_time')}s)")

response \= openai.chat.completions.create(

    model\="gpt-4o",

    messages\=\[

        {"role": "system", "content": "Answer from this knowledge base and cite the passage numbers. " + " ".join(passages)},

        {"role": "user", "content": question},

    \],

)

print(response.choices\[0\].message.content)

### Feature Extractors

Document Graph Extractor

Decompose PDFs into spatial blocks (paragraphs, tables, forms, headers) with layout classification and E5 text embeddings.

Multimodal Extractor

Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.

### Retriever Stages

feature search

Search and filter documents by vector similarity using feature embeddings

filter

## Frequently asked questions

### How does the knowledge base handle different content types together?

All content is embedded into a shared multimodal vector space. Text, video scenes, document pages, and images are all searchable through a single unified query, with results ranked by semantic relevance regardless of format.

### Can I control which content types appear in results?

Yes. Use attribute filters to restrict results to specific content types (e.g., only documents), departments, or access levels.

### How do I keep the knowledge base up to date?

New content uploaded to connected buckets is automatically processed and indexed. Deleted or updated content is reflected in search results without manual reindexing.

## Related code examples and resources

Pages on the same stages, extractors and patterns.

[

Extractor

### Multimodal Extractor

Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.

Learn more





](/extractors/multimodal)[

Extractor

### Document Graph Extractor

Decompose PDFs into spatial blocks (paragraphs, tables, forms, headers) with layout classification and E5 text embeddings.

Learn more





](/extractors/document-graph)[

Recipe

### Metadata Enrichment Pipeline

Automatically enrich your data with extracted metadata: entities, topics, sentiment, language, and custom attributes. Transform raw content into structured, queryable data.

Learn more





](/recipes/metadata-enrichment-pipeline)[

Recipe

### Document RAG Pipeline

Retrieval-augmented generation for document collections. Extracts text, tables, and figures from PDFs using OCR and layout analysis, then retrieves relevant page sections to answer natural language questions with precise page and section citations.

Learn more





](/recipes/document-rag-pipeline)[

Recipe

### Multimodal Hybrid Search Pipeline

Combine vector search with keyword search (BM25) across text, images, and video for the most comprehensive multimodal retrieval system.

Learn more





](/recipes/multimodal-hybrid-search)[

Recipe

### Multimodal RAG Pipeline

Build a retrieval-augmented generation system that works with text, images, and video. Feed relevant multimodal context to LLMs for grounded responses.

Learn more





](/recipes/multimodal-rag-pipeline)