Why Vector Search Alone Can't Find What's in Your Videos
Text-only RAG pipelines miss 80%% of what is in your content. A video contains faces, dialogue, on-screen text, background music, and brand logos. No single embedding captures all of that. The solution is multi-stage retrieval.
TL;DR: Text-only RAG pipelines miss 80% of what's in your content. A video contains faces, dialogue, on-screen text, background music, scene transitions, and brand logos. No single embedding captures all of that. The solution is multi-stage retrieval: extract multiple features per document, search each independently, then merge and rerank the results into one ranked list.
The Problem Everyone Ignores
Most retrieval systems work like this: take content, generate one embedding, store it in a vector database, run cosine similarity at query time. For text documents, this is fine. For everything else, it falls apart.
Consider a 30-second product video. It contains:
- Visual content: product shots, lifestyle imagery, brand colors
- Spoken audio: a voiceover describing features and pricing
- On-screen text: "50% off," a URL, a product name
- Music/tone: upbeat, corporate, dramatic
- Faces: a spokesperson, a customer testimonial
A single CLIP embedding of a keyframe captures maybe the visual content. The dialogue, the on-screen text, the audio tone, the faces? Gone. Your "semantic search" just became a visual-only search that ignores most of the signal.
This is why teams building on video, audio, images, and documents keep hitting the same wall. They get 70% recall and plateau. The missing 30% is cross-modal context that a single embedding cannot represent.
How Retrieval Actually Needs to Work
The fix is not a better embedding model. It is extracting multiple features per document and searching each one independently, then combining the results.
Think of it like a SQL query that JOINs across multiple indices. You would never store a customer's name, purchase history, and support tickets in a single column and expect one query to cover everything. Retrieval over rich media works the same way.
The left side is what most teams build: one model, one embedding, one search. The right side is what production systems need: multiple extractors generating independent feature vectors, independent searches across each, and a merge step that produces one unified ranking.
The Feature Extraction Layer
Before you can search across multiple signals, you need to extract them. This is where most teams get stuck. Running five models per document sounds expensive. It does not have to be.
The key insight is that extraction happens at ingest time, not query time. You pay the compute cost once, then every subsequent search is just a vector lookup. The question is which features to extract.
| Signal | Extractor | What It Captures | Query Example |
|---|---|---|---|
| Visual semantics | CLIP | Scene content, objects, style | "sunset beach product shot" |
| Spoken words | Whisper | Dialogue, narration, speech | "mentions free shipping" |
| On-screen text | PaddleOCR | Titles, captions, URLs, prices | "contains promo code SAVE20" |
| Faces | RetinaFace | Identity, count, position | "video with CEO appearance" |
| Objects | YOLO | Specific items, products, logos | "red Nike shoes" |
| Audio tone | CLAP | Music genre, mood, effects | "upbeat background music" |
| Text meaning | BGE | Semantic content of transcripts | "discusses return policy" |
Each extractor runs independently during ingestion. A single video upload produces 5-7 feature vectors, each queryable on its own. The extraction cost amortizes across every future search.
Multi-Stage Retrieval: The Architecture
Once features are extracted and stored, retrieval becomes a pipeline of stages. Each stage narrows, expands, or reranks the result set. This is what Mixpeek calls a retriever.
A retriever is a sequence of stages that execute in order. Each stage takes the output of the previous stage and transforms it. The stages compose like Unix pipes: each one does one thing, and chaining them produces complex behavior from simple parts.
Stage Types
Search stages query a specific feature index and return candidates:
{
"stage_type": "search",
"model_id": "openai/clip-vit-large-patch14",
"query": { "type": "text", "value": "product demonstration" },
"limit": 100
}Filter stages remove results that do not meet criteria:
{
"stage_type": "filter",
"field": "metadata.duration_seconds",
"operator": "gte",
"value": 15
}Merge stages combine results from parallel searches using reciprocal rank fusion:
{
"stage_type": "merge",
"strategy": "rrf",
"sources": ["visual_search", "transcript_search"]
}Rerank stages re-score the merged results using a cross-encoder or business logic:
{
"stage_type": "rerank",
"method": "weighted",
"weights": { "visual": 0.4, "transcript": 0.35, "ocr": 0.25 }
}The power is in composition. A retriever for "find product videos mentioning free shipping with our CEO" chains: CLIP search for product content + Whisper transcript search for "free shipping" + face search against an enrolled reference collection, merge with RRF, filter by duration, rerank by recency.
Why This Beats Single-Vector Search
The difference is not theoretical. Here are the failure modes that multi-stage retrieval eliminates:
| Scenario | Single Vector | Multi-Stage |
|---|---|---|
| "Find videos where someone says 'quarterly earnings'" | Searches visual embeddings. Returns videos that look like earnings calls. Misses podcast-style recordings. | Searches transcript embeddings. Finds exact phrase regardless of visual content. |
| "Product videos with on-screen pricing" | Returns product videos. Cannot distinguish which ones show prices. | OCR search finds "$" patterns. Intersects with product video filter. |
| "Clips featuring our brand ambassador" | Returns visually similar people. High false positive rate. | Face search against enrolled face collection. Exact identity match. |
| "Upbeat content suitable for social media" | Cannot assess audio tone from visual embedding. | Audio embedding search for "upbeat" + duration filter < 60s. |
Each row is a real query pattern from production deployments. In every case, the multi-stage approach finds results that single-vector search misses entirely, not because the embedding model is bad, but because it is being asked to encode information it was never trained to capture.
The Enrichment Layer: Taxonomies and Clusters
Retrieval is half the story. The other half is enrichment: attaching structured metadata to documents so downstream systems can filter, sort, and categorize without running inference at query time.
Taxonomies work like semantic JOINs. You define a reference collection (brand logos, product SKUs, content categories) and match incoming documents against it using embedding similarity. A video containing a Nike swoosh gets enriched with brand: Nike, brand_id: nike_001, not because a rule detected the text "Nike" but because the visual embedding matched the reference collection.
Clusters work bottom-up. Instead of matching against known categories, clustering groups similar documents and surfaces emergent patterns. You might discover that 40% of your video library shares a visual style you never explicitly categorized.
Together, taxonomies and clusters replace the manual tagging workflows that cost media companies $15-25 per asset.
The Decision Tree: Which Architecture When
Not every use case needs the full multi-stage pipeline. Here is how to decide:
| Your content | Your queries | Start with | Graduate to |
|---|---|---|---|
| Text documents only | Semantic questions | Single embedding + vector search | Add BM25 hybrid search for keyword recall |
| Images with metadata | Visual similarity | CLIP embeddings | Add taxonomy enrichment for structured filters |
| Video (< 1K assets) | Basic search | Scene descriptions | Add transcript + OCR for cross-modal coverage |
| Video (10K+ assets) | Complex, multi-signal | Multi-stage retriever from day one | Add clusters to discover content patterns |
| Mixed media library | Agent-driven queries | MCP integration | Full pipeline: extract, enrich, retrieve, rerank |
The pattern is consistent: start with the simplest pipeline that covers your primary query pattern, then add stages as you discover what the first pipeline misses.
What This Looks Like in Practice
A complete pipeline from upload to searchable, enriched content:
# 1. Upload to a bucket
curl -X POST "$MP_API_URL/v1/buckets/my-bucket/upload" \
-H "Authorization: Bearer $MP_API_KEY" \
-F "[email protected]"
# 2. Collection triggers extraction automatically:
# - CLIP embeddings from video frames
# - Whisper transcription from audio
# - OCR from on-screen text
# - Face detection and embedding
# - Object detection via YOLO
# 3. Taxonomy enrichment runs post-extraction:
# - Matches detected faces against employee collection
# - Matches visual content against brand reference collection
# - Classifies content into IAB categories
# 4. Search across all features at once
curl -X POST "$MP_API_URL/v1/retrievers/my-retriever/search" \
-H "Authorization: Bearer $MP_API_KEY" \
-d '{
"query": {
"text": "product demo with pricing shown on screen",
"modality": "text"
},
"limit": 10
}'The retriever handles the multi-stage logic: parallel searches across visual, transcript, and OCR indices, RRF merge, taxonomy-based filtering, and relevance reranking. The caller sends one query and gets one ranked result list.
The Real Shift
The argument is not that vector search is bad. Vector search is good at what it does: finding semantically similar content within a single modality. The problem is asking it to do everything.
A video is not a text document. An image with overlaid text is not just an image. A podcast episode is not just an audio waveform. Rich media has multiple signals, and each signal needs its own extraction, its own index, and its own search path.
The teams that figure this out stop asking "which embedding model should we use?" and start asking "which features should we extract and how should we combine their search results?" That is the shift from vector search to multimodal retrieval.
Start with one extractor. Add a second when your first query pattern hits a wall. Chain them with a retriever. That is the whole playbook.
Ready to build? Start with the pipeline builder or explore the retriever API reference.