NEWVectors or files. Pick a path.Start →

    Diagrams

    How multimodal search actually works, one picture at a time. Each diagram explains a single idea — from what an embedding is, to why reranking exists, to how the whole extract → index → retrieve loop fits together.

    All (30)
    IR Foundations(12)
    Diagram Series(18)
    The unstructured-data problem: the large majority of enterprise data is unstructured — video, images, audio, and documents — that keyword systems and relational databases cannot search by meaning until it is turned into embeddings and indexed. The raw files are stored but not answerable.
    IR Foundations

    99% of your data has no rows

    Most enterprise data is unstructured — unsearchable by meaning until it is extracted, embedded, and indexed.

    An embedding turns meaning into coordinates: a model reads 'dog', 'puppy', and 'invoice' and outputs vectors, so 'dog' and 'puppy' land near each other while 'invoice' lands far away. Text, images, audio clips, video frames, and faces all map into one shared coordinate space where distance measures difference in meaning.
    IR Foundations

    An embedding is meaning turned into coordinates

    An embedding turns meaning into coordinates — similar things land near each other, across every modality.

    Semantic search as nearest-neighbor lookup: embed the query into the same vector space as your content, measure distance to every indexed point, and return the closest top-K results with no keyword matching. Exact brute-force compares against everything and is accurate but slow, while approximate nearest-neighbor (ANN) indexes like HNSW and IVF check a smart fraction of the space and return 95–99% of true answers in milliseconds.
    IR Foundations

    Search by meaning = nearest neighbors

    Search by meaning is a nearest-neighbor lookup — embed the query, return the closest points.

    Object decomposition: you search inside the file, not for it. A one-hour recording.mp4 is split into scenes (visual embeddings), speech (transcript embeddings on silence and speaker turns), faces (who appeared and when), and on-screen text (OCR of slides and captions). A whole file being findable is not the same as usable — the thing you need is a moment inside it, like 23:41 where the renewal number is spoken while the pricing slide is on screen. The same move works on video, PDFs, and audio: one file becomes dozens of small, independently searchable documents.
    IR Foundations

    You search inside the file, not for it

    You search inside the file, not for it

    Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row — face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.
    IR Foundations

    Pixels in, signals out

    Extraction turns a raw frame into structured, searchable signals — the ceiling for every downstream search.

    Storage versus index: object storage like S3 preserves the raw bytes durably but can never answer a question about what is inside them, while an index reads the content once, extracts what it means, and remembers it so queries get answered in milliseconds without re-reading everything. Without an index, every added terabyte raises the storage bill while the number of answerable questions stays flat at zero.
    IR Foundations

    S3 stores bytes, not answers

    Storage keeps the bytes; an index keeps the meaning — the archive is only useful once it is indexed.

    How you split a document decides what you can find: fixed-size chunks cut mid-sentence and strand the answer across two pieces, while splitting on meaning keeps each chunk a complete thought. Chunk too small and you lose the context that makes an answer usable; chunk too large and the relevant sentence is diluted by everything around it.
    IR Foundations

    How you split decides what you find

    How you split decides what you find

    Hybrid retrieval combines keyword and vector search because each fails alone: keyword (BM25) search misses paraphrases with zero word overlap, while dense embeddings blur exact identifiers like error codes. Mature systems run keyword, vector, and metadata signals together and fuse the ranked lists — reciprocal rank fusion or learned weights — so each signal covers the others' blind spots.
    IR Foundations

    Keywords catch what vectors miss

    Hybrid search fuses keyword (BM25) and vector results so each covers the other's blind spots.

    Reranking is the precision second stage of retrieval: a fast first-stage retriever recalls a broad candidate set optimized for recall, then a cross-encoder reranker reads each query-document pair together and re-scores them so the most relevant results rise to the top before they reach the model. Fast recall first, precise scoring second.
    IR Foundations

    Wide net first, careful read second

    Reranking re-scores first-stage candidates with a cross-encoder so the best results rise to the top.

    Retrieval-augmented generation (RAG): a user asks a question, search retrieves the few most relevant passages from your indexed data, and the model answers from those passages placed in its prompt. The quality ceiling is retrieval, not the model — most hallucinations are retrieval failures, because the model can only answer with what it was fed.
    IR Foundations

    The model answers with what you feed it

    In RAG, retrieval sets the quality ceiling — the model answers from the passages you feed it.

    Choosing a retrieval eval metric is a product decision, not a math one: use Mean Reciprocal Rank (MRR) when users trust the first result (support bots, agents, voice assistants), NDCG@10 when users scan a page of results (product and media search, e-commerce), and recall@k when users need every match (legal discovery, brand safety, compliance). Optimizing the wrong metric makes the dashboard improve while the product degrades, and unmeasured retrieval is not working — it is just not failing loudly yet.
    IR Foundations

    Pick the metric that matches user behavior

    Pick the metric that matches user behavior

    The multimodal data warehouse in one flow: extract signals from media (faces, scenes, speech, on-screen text, embeddings) so pixels become rows at ingest, index that meaning next to the bytes so it is queryable forever after one read, and retrieve by meaning with hybrid search and reranking — served to humans and agents the same way. Extract, index, retrieve; everything else is implementation detail.
    IR Foundations

    Extract, index, retrieve: the whole loop

    Extract, index, retrieve: the whole loop

    A variable-size set of token vectors becomes one fixed-size vector via SimHash bucketing, projection, and repetition — no neural encoder required.
    MUVERA / Multi-Vector

    MUVERA: How a Multi-Vector Set Becomes One Fixed Vector

    How a Multi-Vector Set Becomes One Fixed Vector

    Documents fill empty buckets with a mean; queries sum and leave empty buckets zero — the asymmetry is what makes the FDE dot product approximate Chamfer / MaxSim.
    MUVERA / Multi-Vector

    MUVERA: Why Query and Document Are Encoded Asymmetrically

    Why Query and Document Are Encoded Asymmetrically

    Tier 1 FDE-encodes the query and does fast single-vector ANN over all N to get a top-K; tier 2 reranks only those K with exact Chamfer over the original token vectors.
    MUVERA / Multi-Vector

    MUVERA: Two-Tier Retrieval (Cheap FDE Recall, Exact Rerank)

    Two-Tier Retrieval (Cheap FDE Recall, Exact Rerank)

    Recall@100 versus final FDE dimensionality on nfcorpus with ColBERTv2, log x-axis, one line per swept knob (k_sim, d_proj, r_reps); r_reps dominates at low dimensionality, d_proj floors near 16, and k_sim saturates past 5.
    MUVERA / Multi-Vector

    MUVERA: Which Knob Actually Buys Recall (FDE Dimensionality Ablation)

    Which Knob Actually Buys Recall (FDE Dimensionality Ablation)

    As the mean pairwise cosine of the embeddings rises from 0.26 to 0.99, raw FDE Recall@100 collapses from 0.73 to 0.04, while the mean-centered line stays flat at about 0.725.
    MUVERA / Multi-Vector

    MUVERA: Why Anisotropic Embeddings Break FDEs (and Mean-Centering Fixes It)

    Why Anisotropic Embeddings Break FDEs (and Mean-Centering Fixes It)

    Four chunking strategies applied to the same document: fixed-size cuts mid-sentence, semantic splits on topic shifts, hierarchical nests children under retrievable parents, document-aware respects headings, code blocks, and tables.
    Diagram Series

    Four Chunking Strategies, Same Document

    Four Chunking Strategies, Same Document

    The embedding model landscape: text-only models (E5, BGE, Cohere) for pure text, vision-language models (CLIP, SigLIP, OpenCLIP) for image plus text search, and multimodal models (Vertex 1408D, ImageBind, ONE-PEACE) when video, audio, images, and text must share one space.
    Diagram Series

    The Embedding Model Landscape

    The Embedding Model Landscape

    One ranking, four metrics: with relevant results at positions 1, 3, 7, and 10 — Precision@5 = 0.40, Recall@5 = 0.50, MRR = 1.0, NDCG@10 = 0.83. The right metric depends on how users read your results.
    Diagram Series

    One Ranking, Four Metrics

    One Ranking, Four Metrics

    Multimodal search exposed to an AI agent as an MCP tool: the agent calls a single search tool and Mixpeek runs retrieval across video, images, audio, and documents, returning timestamped, cited results the agent can reason over.
    Diagram Series

    Multimodal Search as an MCP Tool

    Multimodal Search as an MCP Tool

    Video highlight detection pipeline: a video is decomposed into shots and segments, each segment is scored across visual, audio, and transcript signals, contiguous high-scoring intervals merge into moments, and moments are ranked per video.
    Diagram Series

    Multimodal Decomposition: One File, Many Signals

    Multimodal Decomposition: One File, Many Signals

    Multi-vector (late-interaction) search versus a single embedding: instead of compressing a whole document into one vector, the model keeps one vector per token or patch and scores a query against all of them with a MaxSim operator, which recovers detail a single averaged vector loses.
    Diagram Series

    Multi-Vector Search vs. a Single Embedding

    Multi-Vector Search vs. a Single Embedding

    Query expansion rewrites the user's question before retrieval: a short or ambiguous query is expanded into related terms, synonyms, and sub-questions, so first-stage search matches documents the literal phrasing would have missed.
    Diagram Series

    Query Expansion: Rewriting the Question Before You Search

    Query Expansion: Rewriting the Question Before You Search

    How reranking works: a fast bi-encoder recalls ~1,000 candidates from the full corpus, then a cross-encoder jointly scores each (query, document) pair to reorder the top results — fast recall, then precise scoring.
    Diagram Series

    How Reranking Works: Recall Wide, Then Score Precisely

    How Reranking Works: Recall Wide, Then Score Precisely

    The retrieval feedback flywheel: retrieve, interact, learn, improve — clicks, skips, dwell time, and conversions become fusion weights, reranker training pairs, and index partitioning.
    Diagram Series

    The Retrieval Feedback Flywheel

    The Retrieval Feedback Flywheel

    Anatomy of a retriever pipeline: a query flows through staged operations — filtering, first-stage dense or hybrid search, then reranking — with each stage narrowing and reordering candidates before results are returned.
    Diagram Series

    Anatomy of a Retriever Pipeline

    Anatomy of a Retriever Pipeline

    Reciprocal Rank Fusion step by step: dense and BM25 result lists merge by rank position — RRF(d) = sum of 1/(k + rank) with k = 60 — so incompatible score scales never need calibration.
    Diagram Series

    Reciprocal Rank Fusion, Explained

    Reciprocal Rank Fusion, Explained

    Taxonomy-aware retrieval: documents are enriched with taxonomy labels at ingest, so a query can be scoped or boosted by category, letting search combine semantic similarity with a governed classification instead of relying on embeddings alone.
    Diagram Series

    Taxonomy-Aware Retrieval

    Taxonomy-Aware Retrieval

    The three structures behind every vector database: HNSW graphs navigate layered links for high recall, IVF searches only the nearest partitions, PQ compresses vectors roughly 32x — and production systems compose them.
    Diagram Series

    Vector Index Structures: HNSW, IVF, and Friends

    Vector Index Structures: HNSW, IVF, and Friends