The Short Answer
When video search brings back the wrong scene, or the right video at the wrong timestamp, the cause is almost always in how the video was indexed, and rarely in the search itself. Four failures cover most cases:
1. One vector per video. A two-second moment is averaged together with an hour of everything else, so it barely moves the video's embedding. 2. Frames sampled too sparsely. The moment happens between two sampled frames, so nothing in the index ever saw it. 3. One channel only. The index reads pictures and the moment is something someone said, or it reads the transcript and the moment is something on screen. 4. Queries embedded differently from the content. Every score drops and the order shuffles, so the right scene lands a few places down.
Index per scene, cut at scene boundaries, search the picture, the speech and the on-screen text together, and embed queries the way the content was embedded.
Why one vector per video loses the moment
The simplest video index embeds a handful of frames and averages them into one vector per file. That works for "find videos about cooking" and fails for "find the moment the pan catches fire", because the fire is two seconds of a forty-minute video and contributes about one part in a thousand to the average.
The symptom is specific: the right video tends to come back, and the timestamp is wherever the video starts, because the index never knew where anything happened.
The fix is to change the unit you index. Split the video into scenes and give each scene its own entry, with its start and end time. Video scene segmentation covers shot detection and scene grouping. A search then ranks scenes, and the timestamp comes for free.
Why the moment falls between frames
Sampling one frame every few seconds is a common default, and it quietly drops anything shorter than the gap. A slate, a logo flash, a hand-off, a product shown for a second: none of it is in the index if no sampled frame landed on it.
Fixed intervals also waste frames on long static shots and starve fast cuts. Sampling at scene boundaries, plus a few frames inside long scenes, spends the same budget where the content changes. Video frame sampling covers how many frames to keep and which ones.
A quick test tells you whether this is your problem: take a query that fails, find the right moment by hand, and check whether any indexed frame falls inside it. If none does, no ranking change will ever surface it.
Why the index cannot see what was said, or what was written
A frame embedding knows what a scene looks like. It does not know that someone said "we are recalling the product" over a shot of an office, and it cannot read a phone number on a lower third. A transcript has the opposite blind spot.
So a query about speech run against an image-only index returns scenes that look related and say nothing relevant, and a query about a sign or a caption returns nothing close. The fix is to index each scene several ways (its frames, its transcript segment, and the text detected on screen) and search them together. Why video search misses on-screen text covers the caption case, and finding the moment in a podcast covers the speech case.
Why the right scene ranks fourth
Many embedding models expect a query and a document to be encoded differently. Some prepend an instruction to queries, some expect a prefix such as "query:" on one side and "passage:" on the other, and some expect no prefix at all. If your scenes were embedded one way and your queries another, the right scene still scores high, only slightly lower than it should, and a few neighbours overtake it.
This one is easy to miss because nothing looks broken. Check how the content was embedded, then embed queries to match. Mixed indexes are the usual source: vectors imported from one pipeline and queried through another.
A second-stage reranker also helps here, because it compares the query with each candidate directly instead of trusting the first-pass score. Cross-encoder reranking covers the pattern.
How to tell which problem you have
Take twenty queries where you know the right video and the right moment, and write down two ranks for each: where the right video lands, and where the right scene lands.
| What you see | Most likely cause | First fix |
| Right video, timestamp at the start or random | One vector per video | Index per scene |
| Right moment never appears at any rank | Sampling missed it | Sample at scene boundaries |
| Spoken or on-screen queries fail, visual ones work | One channel only | Add transcript and on-screen text |
| Right scene usually in the top ten, rarely first | Query/content embedding mismatch | Match the query encoding, add a reranker |
Why raising the similarity threshold does not help
Cutting off low scores removes noise and does nothing for a moment that was never indexed or was averaged away. Scores also are not comparable across models or queries, so a cutoff tuned on one query drops good matches on another. Calibrating similarity scores covers what the number means.
Doing this on Mixpeek
Mixpeek's video extractors index per scene. They split each video at scene boundaries, embed the frames, transcribe the speech and read the text on screen, and store each scene as its own document with its start and end time. A retriever searches those channels together and can rerank before it returns, so a query comes back as scenes with timestamps. It runs over the video already in your S3 or GCS bucket.
A text query can also set the embedding instruction it is encoded with, which fixes the ranking drift above when your vectors came from another pipeline.
See it working on the footage search template, read how the video search API returns timestamps, and compare costs on pricing. If you already have your own video embeddings, MVS searches them on object storage.
Frequently Asked Questions
Why does my video search return the right video but the wrong timestamp?
The index most likely holds one entry per video, so it can say which video matches and nothing about where. Index each scene separately with its own start and end time and the timestamp comes from the scene that matched.
How often should I sample frames from a video for search?
Sample at scene boundaries rather than on a fixed clock, and add a few frames inside long scenes. A fixed interval misses anything shorter than the gap and spends frames on shots where nothing changes.
Can video search find something that is only said, never shown?
Only if the speech is indexed. Transcribe the audio, keep the timestamps, and search the transcript alongside the frames. A frame-only index cannot find it at any rank.
Would a better embedding model fix wrong-scene results?
It helps with close calls and does nothing for a moment that was averaged away or never sampled. Fix the index unit and the sampling first, then compare models.
Why do my results change order when I import vectors from another tool?
The imported vectors were probably embedded with a different query or document instruction than the one your search uses. Encode queries the same way the content was encoded and the order settles.