NEWVectors or files. Pick a path.Start →
    Back to DiagramsVideo Understanding

    Frame Sampling: Which Frames Get Embedded, and What It Costs

    Embedding is the expensive step, so the frames you choose to embed are the biggest lever on the cost and recall of a video search system.

    Frame sampling decides which frames of a video get embedded, the single biggest lever on cost and recall. An hour of 30 fps video is 108,000 frames, and adjacent frames within a shot are near identical, so their embeddings land almost on top of each other and indexing both adds storage and query fan-out, not recall. The same timeline (a 3 minute lecture slide, then a 4 second chaotic cut) sampled three ways: fixed rate 1 fps embeds 184 frames and still skips half the shots in the burst, a scene-adaptive budget covers everything with 9, and embedding-space keyframe selection keeps 6 chosen for visual diversity. The real output is the recall versus cost frontier: plot recall against vectors stored per video-hour and find the knee for your content, per query class.
    Embedding is the expensive step, so the frames you choose to embed are the biggest lever on the cost and recall of a video search system.

    An hour of 30 fps video is 108,000 frames, and adjacent frames within a shot are near identical: their embeddings land almost on top of each other, so indexing both adds storage and query fan-out, not recall. Embedding is the expensive step, which makes sampling policy the single biggest lever on the cost and recall of a video search system. A sampling change can move the cost line more than switching models does.

    Take one timeline: a 3 minute lecture slide followed by a 4 second chaotic cut. Fixed-rate sampling at 1 fps spends the same budget everywhere: it embeds 184 frames and still skips half the shots in the burst, because it pays for recall where nothing is happening and starves it where everything is. A scene-adaptive budget places a floor of 1 to 2 frames on quiet stretches and a cap of about 8 on busy ones, covering the whole clip with 9 frames. Embedding-space keyframe selection goes further: it generates cheap dense candidates, then keeps a handful (6 here) chosen for visual diversity rather than by the clock.

    The honest output of a sampling study is not a single number, it is a frontier. Plot recall against vectors stored per video-hour and look for the knee, and measure it per query class, because the right sampling rate for a static screen recording is the wrong one for sports. Cache frames and embeddings by content hash, or a re-run silently re-pays the full embedding bill.

    Where this diagram appears

    Run this on your own data

    Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

    Search your own data, free

    Related diagrams

    An embedding turns meaning into coordinates: a model reads 'dog', 'puppy', and 'invoice' and outputs vectors, so 'dog' and 'puppy' land near each other while 'invoice' lands far away. Text, images, audio clips, video frames, and faces all map into one shared coordinate space where distance measures difference in meaning.

    An embedding is meaning turned into coordinates

    Object decomposition: you search inside the file, not for it. A one-hour recording.mp4 is split into scenes (visual embeddings), speech (transcript embeddings on silence and speaker turns), faces (who appeared and when), and on-screen text (OCR of slides and captions). A whole file being findable is not the same as usable: the thing you need is a moment inside it, like 23,41 where the renewal number is spoken while the pricing slide is on screen. The same move works on video, PDFs, and audio: one file becomes dozens of small, independently searchable documents.

    You search inside the file, not for it

    Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row, face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.

    Pixels in, signals out