NEWVectors or files. Pick a path.Start →
    Back to DiagramsVideo Understanding

    Video RAG: Why Retrieval Over Video Is Not Retrieval Over Text

    Text RAG gets chunk boundaries, one modality, and small units for free. Video gives you none of them, so every stage of the pipeline is redesigned.

    Video RAG redesigns every stage of retrieval-augmented generation because video breaks the three assumptions text RAG relies on: natural chunk boundaries (paragraphs), a uniform modality (a chunk is always text), and small retrieval units (a few hundred tokens). Video is a continuous stream with no inherent segmentation, a single segment carries frames, speech, background sound, and on-screen text at once, and ten seconds at 30 fps is already 300 frames. So chunking becomes scene segmentation (TransNetV2, PySceneDetect, scene chunks of 30 seconds to 5 minutes), one index becomes dual-channel indexing (visual embeddings from keyframes plus a Whisper transcript aligned to timestamps), retrieval becomes multi-stage (ANN pulls 50 to 100 candidates, cross-modal reranking cuts to 10 to 20, temporal grounding pinpoints the moment), context assembly becomes frame selection (about 16 frames per scene, temporally ordered, diminishing returns past 32 to 64), and the citation becomes a timestamp the user can click to land on the exact moment.
    Text RAG gets chunk boundaries, one modality, and small units for free. Video gives you none of them, so every stage of the pipeline is redesigned.

    Text RAG works because text hands you three things for free: natural chunk boundaries at paragraphs and sections, a uniform modality where a chunk is always text, and small retrieval units of a few hundred tokens that drop straight into a prompt. Video gives you none of them. It is a continuous stream with no inherent segmentation, a single segment carries frames, speech, background sound, and on-screen text at once, and ten seconds at 30 fps is already 300 full-resolution images.

    So every stage of the pipeline gets redesigned. Chunking becomes scene segmentation, using shot boundary detection (TransNetV2, PySceneDetect) plus semantic grouping into scene-level chunks of 30 seconds to 5 minutes. One index becomes dual-channel indexing: visual embeddings from keyframes (CLIP, SigLIP) alongside an ASR transcript (Whisper) aligned to timestamps and embedded as text. Retrieval becomes multi-stage: ANN pulls 50 to 100 candidates, cross-modal reranking cuts to 10 to 20, and temporal grounding pinpoints the exact start and end inside a scene. Context assembly becomes frame selection, since a 2 minute scene is 3,600 frames and a large vision-language model wants about 16 of them, temporally ordered, with diminishing returns past 32 to 64.

    The payoff is the last stage. The answer cites the source scene and a timestamp, and the user clicks it to jump to the exact moment. The retrieved moments carry their timestamps all the way into the answer, which is the thing text RAG never had to think about and video RAG is built around.

    Where this diagram appears

    Run this on your own data

    Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

    Search your own data, free

    Related diagrams

    Object decomposition: you search inside the file, not for it. A one-hour recording.mp4 is split into scenes (visual embeddings), speech (transcript embeddings on silence and speaker turns), faces (who appeared and when), and on-screen text (OCR of slides and captions). A whole file being findable is not the same as usable: the thing you need is a moment inside it, like 23,41 where the renewal number is spoken while the pricing slide is on screen. The same move works on video, PDFs, and audio: one file becomes dozens of small, independently searchable documents.

    You search inside the file, not for it

    Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row, face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.

    Pixels in, signals out

    How you split a document decides what you can find: fixed-size chunks cut mid-sentence and strand the answer across two pieces, while splitting on meaning keeps each chunk a complete thought. Chunk too small and you lose the context that makes an answer usable; chunk too large and the relevant sentence is diluted by everything around it.

    How you split decides what you find