NEWVectors or files. Pick a path.Start →

    Diagrams

    How multimodal search actually works, one picture at a time. Each diagram explains a single idea: from what an embedding is, to why reranking exists, to how the whole extract → index → retrieve loop fits together.

    All (48)
    IR Foundations(16)
    Diagram Series(32)
    One article URL producing five structured signals: IAB category at tier 1 and tier 2, main entities each carrying a salience score from 0 to 1 drawn as a ranked bar chart, article-level sentiment, a brand-safety flag with categories, and keywords. A side panel contrasts a flat entity list against the same entities scored.
    Retrieval Mechanics

    Contextual Page Signals: Scoring Entities Instead of Listing Them

    Nine entities come off one local-news page, three of them US politicians. Flat, it reads as a politics page. Scored, the top two are the actress and the sitcom and it is obviously a television story.

    Diagram of filtered vector search showing how filter selectivity decides which of three strategies works: pre-filtering, post-filtering, and single-stage filtered traversal, with the failure each one hits at the wrong selectivity.
    Retrieval Mechanics

    Filtered Vector Search: Why the Same Filter Is Free or Fatal

    Adding a filter to a vector search looks like a WHERE clause. Selectivity decides whether it costs nothing or destroys your results, and the API call is identical either way.

    Diagram placing BM25, SPLADE and dense embeddings on a single axis measuring how much vocabulary survives compression, with what each one stores and why production systems run more than one.
    Retrieval Mechanics

    Sparse and Dense Are One Spectrum, Not Two Camps

    BM25, SPLADE and dense embeddings sit on one axis. What moves along it is how much vocabulary survives compression, which is why a rare identifier ranks below three passages that never mention it.

    Diagram contrasting a human query with an agent query, showing how agent calls carry accumulated state and prior tool output, and what a retrieval layer must return differently as a result.
    Retrieval Mechanics

    Agentic Retrieval: What Changes When the Caller Is Not a Person

    A person types four words. An agent sends its accumulated state, prior tool output and the reason it is asking. The retrieval layer is answering a different kind of caller.

    Diagram comparing memory-resident vector search against storage-first vector search on object storage, showing the cost structure of a billion 768-dimensional vectors under each model.
    Architecture

    The S3 Vector Warehouse: Storage-First Vector Search

    A billion 768-dimensional float32 vectors is about 3TB raw. Memory-resident, that is a cluster running whether anyone queries it or not. On object storage it is a rounding error of monthly storage.

    Diagram showing why a naive for-each ingestion loop fails at scale, and the checkpointing, isolation and retry structure that replaces it.
    Architecture

    Ingestion at Scale: Why the Obvious Loop Dies at a Million Files

    The obvious loop is correct for a thousand files. At a million, one corrupt video kills it with 400,000 files unprocessed and no record of where it stopped.

    Market map of 27 object storage providers across seven categories (hyperscalers, zero-egress, low-cost independents, bundled developer clouds, decentralized networks, self-hosted software, enterprise appliances), each showing storage cost per TB-month, egress cost per TB, and ingress cost, with a closing panel pricing the cost of moving 100 TB out of each major provider.
    Market Maps

    The Object Storage Market Map: 27 Providers, Priced

    27 object storage providers across seven categories, each carrying storage $/TB-month, egress $/TB, and ingress. The closing panel prices what it costs to walk out with 100 TB.

    Contrastive learning trains a model to answer one question: are these two things related? You give it a batch of N anchor-positive pairs (an image and its own caption for CLIP, an audio clip and its description for CLAP) and nobody ever labels how similar a pair is, because the pairing itself is the entire supervision signal. Inside one batch each anchor has one positive and N-1 negatives, since every other item is by definition a non-match. CLIP scores the batch as an N by N grid of cosine similarities where the diagonal holds the true pairings, and InfoNCE pulls the diagonal together while pushing everything off it apart. Batch size is therefore a first-class hyperparameter rather than a throughput knob: CLIP trained at 32,768 so each anchor faced 32,767 simultaneous negatives and could not win on coarse cues. CLIP uses a softmax over the batch, SigLIP replaces it with a pairwise sigmoid that needs no global normalization and so trains stably at large scale.
    IR Foundations

    Contrastive Learning: How Two Encoders Learn One Space

    Why a text query can retrieve an image: the pairing is the only supervision.

    Train a dual encoder like CLIP and you would expect a matched image and its caption to land at nearly the same point in the shared space. They do not. All image embeddings occupy one narrow cone of the hypersphere and all text embeddings occupy a different, well-separated cone, and the two clouds barely overlap. A matched pair is closer to each other than to mismatched pairs, but both still sit inside their own modality's region, separated by a roughly constant offset vector. Three forces create and preserve it: a dual encoder runs each modality through a separate network with no shared weights, InfoNCE only requires a matched pair to outscore in-batch negatives which is satisfied with every image in cone A and every caption in cone B, and at initialization each encoder's outputs already occupy a small cone while a low softmax temperature sharpens matched against unmatched pairs without pulling the cones together. The model was trained to rank, not to register.
    IR Foundations

    The Modality Gap: A Shared Space Is Really Two Cones

    Why cross-modal retrieval underperforms same-modality retrieval in one shared space.

    Every relevance score in retrieval comes from one of three architectures, and they differ on exactly one axis: when the query is allowed to see the document. A bi-encoder never lets them meet until the end, encoding query and document separately into one vector each and scoring by cosine, so document vectors are computed once at index time and search is an ANN lookup, but the encoder had to guess what mattered before the question existed. A cross-encoder lets them meet immediately, concatenating query and document so every query token attends to every document token, which is worth 5 to 15% on benchmarks but costs quadratic time per pair and can only rerank a shortlist. Late interaction keeps a vector per token, computes the interaction at query time with MaxSim (each query token takes its best match in the document, then sum), and pays for it in storage: hundreds of vectors per document instead of one.
    MUVERA / Multi-Vector

    Three Ways to Score a Query Against a Document

    Bi-encoder, cross-encoder, late interaction: when the query is allowed to see the document.

    Vector memory is one multiplication: bytes equals vectors times dimensions times bytes per dimension. At 1,024 dimensions in float32 that is 4,096 bytes per vector, so 10 million vectors is 40.96 GB and a billion is 4.1 TB. The relationship is linear, which means there are exactly three levers and every compression technique is a discount on one of them. int8 quantization scales each dimension into a byte for a 4x cut, binary quantization keeps only the sign of each dimension for 32x, product quantization splits the vector into subvectors and replaces each with a centroid id for arbitrary ratios, and Matryoshka truncates dimensions because the model was trained so the first d dimensions are themselves a valid embedding. The rescore cascade is what makes any of it survivable: retrieve wide with the cheap representation, then re-rank the shortlist with a more precise one.
    IR Foundations

    The Memory Math of Quantization

    Every compression method is a discount on one term of vectors x dimensions x bytes.

    A similarity score is an ordinal signal. Within one query, against one index, with one model, higher is better, and that ordering is the entire guarantee. A cosine of 0.80 does not mean 80% similar, and the score is not linear in relevance either: the step from 0.9 to 0.8 is not the same loss as the step from 0.4 to 0.3. Four independent effects shape the absolute value and none is about whether the result is relevant. Training temperature: contrastive models divide similarity by a temperature, so two models can rank identically while one reports around 0.9 and the other around 0.4. The modality gap: a 0.3 text-to-image match can be as strong as a 0.7 image-to-image one. Distribution shape: some models cluster everything between 0.25 and 0.45, where 0.45 is the best possible match. And per-query difficulty: a rare query scores lower even when the right answer is first.
    IR Foundations

    A 0.83 Cosine Is Not 83% Relevant

    Scores are ordinal, not cardinal, which is why a fixed threshold silently breaks.

    The unstructured-data problem: the large majority of enterprise data is unstructured (video, images, audio, and documents) that keyword systems and relational databases cannot search by meaning until it is turned into embeddings and indexed. The raw files are stored but not answerable.
    IR Foundations

    99% of your data has no rows

    Most enterprise data is unstructured: unsearchable by meaning until it is extracted, embedded, and indexed.

    An embedding turns meaning into coordinates: a model reads 'dog', 'puppy', and 'invoice' and outputs vectors, so 'dog' and 'puppy' land near each other while 'invoice' lands far away. Text, images, audio clips, video frames, and faces all map into one shared coordinate space where distance measures difference in meaning.
    IR Foundations

    An embedding is meaning turned into coordinates

    An embedding turns meaning into coordinates: similar things land near each other, across every modality.

    Semantic search as nearest-neighbor lookup: embed the query into the same vector space as your content, measure distance to every indexed point, and return the closest top-K results with no keyword matching. Exact brute-force compares against everything and is accurate but slow, while approximate nearest-neighbor (ANN) indexes like HNSW and IVF check a smart fraction of the space and return 95-99% of true answers in milliseconds.
    IR Foundations

    Search by meaning = nearest neighbors

    Search by meaning is a nearest-neighbor lookup: embed the query, return the closest points.

    Object decomposition: you search inside the file, not for it. A one-hour recording.mp4 is split into scenes (visual embeddings), speech (transcript embeddings on silence and speaker turns), faces (who appeared and when), and on-screen text (OCR of slides and captions). A whole file being findable is not the same as usable: the thing you need is a moment inside it, like 23,41 where the renewal number is spoken while the pricing slide is on screen. The same move works on video, PDFs, and audio: one file becomes dozens of small, independently searchable documents.
    IR Foundations

    You search inside the file, not for it

    You search inside the file, not for it

    Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row, face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.
    IR Foundations

    Pixels in, signals out

    Extraction turns a raw frame into structured, searchable signals: the ceiling for every downstream search.

    Storage versus index: object storage like S3 preserves the raw bytes durably but can never answer a question about what is inside them, while an index reads the content once, extracts what it means, and remembers it so queries get answered in milliseconds without re-reading everything. Without an index, every added terabyte raises the storage bill while the number of answerable questions stays flat at zero.
    IR Foundations

    S3 stores bytes, not answers

    Storage keeps the bytes; an index keeps the meaning: the archive is only useful once it is indexed.

    How you split a document decides what you can find: fixed-size chunks cut mid-sentence and strand the answer across two pieces, while splitting on meaning keeps each chunk a complete thought. Chunk too small and you lose the context that makes an answer usable; chunk too large and the relevant sentence is diluted by everything around it.
    IR Foundations

    How you split decides what you find

    How you split decides what you find

    Hybrid retrieval combines keyword and vector search because each fails alone: keyword (BM25) search misses paraphrases with zero word overlap, while dense embeddings blur exact identifiers like error codes. Mature systems run keyword, vector, and metadata signals together and fuse the ranked lists, reciprocal rank fusion or learned weights, so each signal covers the others' blind spots.
    IR Foundations

    Keywords catch what vectors miss

    Hybrid search fuses keyword (BM25) and vector results so each covers the other's blind spots.

    Reranking is the precision second stage of retrieval: a fast first-stage retriever recalls a broad candidate set optimized for recall, then a cross-encoder reranker reads each query-document pair together and re-scores them so the most relevant results rise to the top before they reach the model. Fast recall first, precise scoring second.
    IR Foundations

    Wide net first, careful read second

    Reranking re-scores first-stage candidates with a cross-encoder so the best results rise to the top.

    Retrieval-augmented generation (RAG): a user asks a question, search retrieves the few most relevant passages from your indexed data, and the model answers from those passages placed in its prompt. The quality ceiling is retrieval, not the model: most hallucinations are retrieval failures, because the model can only answer with what it was fed.
    IR Foundations

    The model answers with what you feed it

    In RAG, retrieval sets the quality ceiling: the model answers from the passages you feed it.

    Choosing a retrieval eval metric is a product decision, not a math one: use Mean Reciprocal Rank (MRR) when users trust the first result (support bots, agents, voice assistants), NDCG@10 when users scan a page of results (product and media search, e-commerce), and recall@k when users need every match (legal discovery, brand safety, compliance). Optimizing the wrong metric makes the dashboard improve while the product degrades, and unmeasured retrieval is not working: it is just not failing loudly yet.
    IR Foundations

    Pick the metric that matches user behavior

    Pick the metric that matches user behavior

    The multimodal data warehouse in one flow: extract signals from media (faces, scenes, speech, on-screen text, embeddings) so pixels become rows at ingest, index that meaning next to the bytes so it is queryable forever after one read, and retrieve by meaning with hybrid search and reranking, served to humans and agents the same way. Extract, index, retrieve; everything else is implementation detail.
    IR Foundations

    Extract, index, retrieve: the whole loop

    Extract, index, retrieve: the whole loop

    A variable-size set of token vectors becomes one fixed-size vector via SimHash bucketing, projection, and repetition: no neural encoder required.
    MUVERA / Multi-Vector

    MUVERA: How a Multi-Vector Set Becomes One Fixed Vector

    How a Multi-Vector Set Becomes One Fixed Vector

    Documents fill empty buckets with a mean; queries sum and leave empty buckets zero: the asymmetry is what makes the FDE dot product approximate Chamfer / MaxSim.
    MUVERA / Multi-Vector

    MUVERA: Why Query and Document Are Encoded Asymmetrically

    Why Query and Document Are Encoded Asymmetrically

    Tier 1 FDE-encodes the query and does fast single-vector ANN over all N to get a top-K; tier 2 reranks only those K with exact Chamfer over the original token vectors.
    MUVERA / Multi-Vector

    MUVERA: Two-Tier Retrieval (Cheap FDE Recall, Exact Rerank)

    Two-Tier Retrieval (Cheap FDE Recall, Exact Rerank)

    Recall@100 versus final FDE dimensionality on nfcorpus with ColBERTv2, log x-axis, one line per swept knob (k_sim, d_proj, r_reps); r_reps dominates at low dimensionality, d_proj floors near 16, and k_sim saturates past 5.
    MUVERA / Multi-Vector

    MUVERA: Which Knob Actually Buys Recall (FDE Dimensionality Ablation)

    Which Knob Actually Buys Recall (FDE Dimensionality Ablation)

    As the mean pairwise cosine of the embeddings rises from 0.26 to 0.99, raw FDE Recall@100 collapses from 0.73 to 0.04, while the mean-centered line stays flat at about 0.725.
    MUVERA / Multi-Vector

    MUVERA: Why Anisotropic Embeddings Break FDEs (and Mean-Centering Fixes It)

    Why Anisotropic Embeddings Break FDEs (and Mean-Centering Fixes It)

    Four chunking strategies applied to the same document: fixed-size cuts mid-sentence, semantic splits on topic shifts, hierarchical nests children under retrievable parents, document-aware respects headings, code blocks, and tables.
    Diagram Series

    Four Chunking Strategies, Same Document

    Four Chunking Strategies, Same Document

    The embedding model landscape: text-only models (E5, BGE, Cohere) for pure text, vision-language models (CLIP, SigLIP, OpenCLIP) for image plus text search, and multimodal models (Vertex 1408D, ImageBind, ONE-PEACE) when video, audio, images, and text must share one space.
    Diagram Series

    The Embedding Model Landscape

    The Embedding Model Landscape

    One ranking, four metrics: with relevant results at positions 1, 3, 7, and 10, Precision@5 = 0.40, Recall@5 = 0.50, MRR = 1.0, NDCG@10 = 0.83. The right metric depends on how users read your results.
    Diagram Series

    One Ranking, Four Metrics

    One Ranking, Four Metrics

    Multimodal search exposed to an AI agent as an MCP tool: the agent calls a single search tool and Mixpeek runs retrieval across video, images, audio, and documents, returning timestamped, cited results the agent can reason over.
    Diagram Series

    Multimodal Search as an MCP Tool

    Multimodal Search as an MCP Tool

    Video highlight detection pipeline: a video is decomposed into shots and segments, each segment is scored across visual, audio, and transcript signals, contiguous high-scoring intervals merge into moments, and moments are ranked per video.
    Diagram Series

    Multimodal Decomposition: One File, Many Signals

    Multimodal Decomposition: One File, Many Signals

    Multi-vector (late-interaction) search versus a single embedding: instead of compressing a whole document into one vector, the model keeps one vector per token or patch and scores a query against all of them with a MaxSim operator, which recovers detail a single averaged vector loses.
    Diagram Series

    Multi-Vector Search vs. a Single Embedding

    Multi-Vector Search vs. a Single Embedding

    Query expansion rewrites the user's question before retrieval: a short or ambiguous query is expanded into related terms, synonyms, and sub-questions, so first-stage search matches documents the literal phrasing would have missed.
    Diagram Series

    Query Expansion: Rewriting the Question Before You Search

    Query Expansion: Rewriting the Question Before You Search

    How reranking works: a fast bi-encoder recalls ~1,000 candidates from the full corpus, then a cross-encoder jointly scores each (query, document) pair to reorder the top results, fast recall, then precise scoring.
    Diagram Series

    How Reranking Works: Recall Wide, Then Score Precisely

    How Reranking Works: Recall Wide, Then Score Precisely

    The retrieval feedback flywheel: retrieve, interact, learn, improve, clicks, skips, dwell time, and conversions become fusion weights, reranker training pairs, and index partitioning.
    Diagram Series

    The Retrieval Feedback Flywheel

    The Retrieval Feedback Flywheel

    Anatomy of a retriever pipeline: a query flows through staged operations (filtering, first-stage dense or hybrid search, then reranking) with each stage narrowing and reordering candidates before results are returned.
    Diagram Series

    Anatomy of a Retriever Pipeline

    Anatomy of a Retriever Pipeline

    Reciprocal Rank Fusion step by step: dense and BM25 result lists merge by rank position, RRF(d) = sum of 1/(k + rank) with k = 60, so incompatible score scales never need calibration.
    Diagram Series

    Reciprocal Rank Fusion, Explained

    Reciprocal Rank Fusion, Explained

    Taxonomy-aware retrieval: documents are enriched with taxonomy labels at ingest, so a query can be scoped or boosted by category, letting search combine semantic similarity with a governed classification instead of relying on embeddings alone.
    Diagram Series

    Taxonomy-Aware Retrieval

    Taxonomy-Aware Retrieval

    The three structures behind every vector database: HNSW graphs navigate layered links for high recall, IVF searches only the nearest partitions, PQ compresses vectors roughly 32x, and production systems compose them.
    Diagram Series

    Vector Index Structures: HNSW, IVF, and Friends

    Vector Index Structures: HNSW, IVF, and Friends

    Reverse image search embeds one image and returns nearest neighbors: it matches a point in embedding space. A video clip embeds frame by frame into a trajectory through that same space, so reverse video search means finding another video whose path lines up with yours for seconds at a time, and the aligned segment carries its timestamps. The four-stage pipeline samples frames, represents them as perceptual fingerprints or semantic embeddings, indexes them with (video_id, timestamp) payloads, and localizes the matching moment. The match survives re-encodes, crops, letterboxing, overlays, and re-cuts because the pixels change but the path keeps its shape.
    Video Retrieval

    Reverse Video Search: A Clip Is a Trajectory, Not a Point

    Why image reverse-search matches a point and video reverse-search matches a path.

    Scene segmentation turns a continuous video into searchable segments in two levels. First, shot boundary detection compares consecutive frames: a hard cut spikes a histogram difference (over 1000 fps on CPU), while dissolves and fades spread the change across many frames, so adaptive detectors flag a boundary when the score beats a rolling local mean by 2 to 3 sigma. Second, shots group into scenes, where a shot is one continuous camera take and a scene is a semantic unit that can span many shots. Cleanup rules merge segments under 2 seconds into their nearest neighbor and split segments over 60 seconds at audio silence. The scene becomes the retrieval unit: one embedding per scene instead of per frame, results that carry start and end timestamps, and storage down 10 to 100x.
    Video Understanding

    Scene Segmentation: How a Video Becomes Searchable Scenes

    A raw video has no scenes. Two levels of boundary detection turn 108,000 frames an hour into a handful of searchable segments.

    Frame sampling decides which frames of a video get embedded, the single biggest lever on cost and recall. An hour of 30 fps video is 108,000 frames, and adjacent frames within a shot are near identical, so their embeddings land almost on top of each other and indexing both adds storage and query fan-out, not recall. The same timeline (a 3 minute lecture slide, then a 4 second chaotic cut) sampled three ways: fixed rate 1 fps embeds 184 frames and still skips half the shots in the burst, a scene-adaptive budget covers everything with 9, and embedding-space keyframe selection keeps 6 chosen for visual diversity. The real output is the recall versus cost frontier: plot recall against vectors stored per video-hour and find the knee for your content, per query class.
    Video Understanding

    Frame Sampling: Which Frames Get Embedded, and What It Costs

    Embedding is the expensive step, so the frames you choose to embed are the biggest lever on the cost and recall of a video search system.

    Temporal grounding takes a natural language query and a video and returns the start and end timestamps of the described event, turning the video from an opaque blob into a queryable timeline. Frame-level similarity search only returns stills: the best it can say is that one frame looks relevant, with no beginning or end. The core loop is per-segment scoring plus interval merging: sample frames at 1 fps, embed frames and query with the same model (CLIP or SigLIP), score every frame against the query, slide a window and average the scores, then merge the overlapping windows that clear a threshold into one segment with a start and an end (for example start 134.2, end 141.8, score 0.94). Fancier methods keep the same contract: proposal networks like 2D-TAN and Moment-DETR learn the boundaries, dense captioning turns grounding into text search, and production systems decompose the video into timestamped feature streams and intersect intervals with temporal joins. The primary metric is Recall at 1 with IoU 0.5.
    Video Understanding

    Temporal Grounding: Query In, Timestamp Out

    Frame search returns stills. Temporal grounding takes a text query and a video and returns the start and end timestamps of the moment you asked for.

    Video RAG redesigns every stage of retrieval-augmented generation because video breaks the three assumptions text RAG relies on: natural chunk boundaries (paragraphs), a uniform modality (a chunk is always text), and small retrieval units (a few hundred tokens). Video is a continuous stream with no inherent segmentation, a single segment carries frames, speech, background sound, and on-screen text at once, and ten seconds at 30 fps is already 300 frames. So chunking becomes scene segmentation (TransNetV2, PySceneDetect, scene chunks of 30 seconds to 5 minutes), one index becomes dual-channel indexing (visual embeddings from keyframes plus a Whisper transcript aligned to timestamps), retrieval becomes multi-stage (ANN pulls 50 to 100 candidates, cross-modal reranking cuts to 10 to 20, temporal grounding pinpoints the moment), context assembly becomes frame selection (about 16 frames per scene, temporally ordered, diminishing returns past 32 to 64), and the citation becomes a timestamp the user can click to land on the exact moment.
    Video Understanding

    Video RAG: Why Retrieval Over Video Is Not Retrieval Over Text

    Text RAG gets chunk boundaries, one modality, and small units for free. Video gives you none of them, so every stage of the pipeline is redesigned.

    Speaker diarization answers who spoke and when, the layer transcription alone cannot provide. The same sixteen seconds of a meeting are shown at three levels: the raw waveform, voice activity detection marking speech regions, and clustered speaker turns, producing a transcript whose every line carries a speaker label and a timestamp. The standard pipeline is four stages: voice activity detection (Silero VAD, a 2MB model running 500x real-time on CPU at about 95% accuracy), speaker embedding extraction (ECAPA-TDNN, 192 or 256 dimensions over 1.5 to 3 second sliding windows, capturing pitch and timbre regardless of words), spectral clustering that reads the speaker count from the eigenvalue gap, and overlap detection that allows multiple labels on one region. Diarization is then joined with ASR word timestamps to attribute each word, and the field scores itself with diarization error rate (false alarm plus missed speech plus speaker confusion over total speech), where under 10% is production quality.
    Video Understanding

    Speaker Diarization: Who Said What, and When

    Transcription tells you what was said. Diarization tells you who said it and when, by clustering voices before it ever aligns them to the transcript.