How Reverse Video Search Works: Embeddings, Time, and Copy Detection

VSC23 formalizes the three moves that any production reverse video search has to make, and it is worth understanding them separately because each maps to a different part of a retrieval pipeline. First, *describe every second*. The challenge caps descriptors at 512 dimensions per second of video, compared by inner product. That constraint is the interesting part: it rules out the tempting shortcut of collapsing a whole video into one embedding, because a copied 8-second clip inside a 40-minute video would be averaged into invisibility. Second, *align in time*. Detection alone is not useful to a rights or editorial team — you need to know which seconds match, so the descriptor sequences are aligned to recover the shared segment with timecodes. Third, *train the descriptor on copies*. Self-supervised descriptors in the SSCD line (arXiv:2202.10261) are trained so that an edited version of an image lands next to its original, which is what makes matches survive crops, re-encodes, speed changes, and overlays. The benchmark numbers are the argument for doing all three: a strong image descriptor applied on its own reaches 60.5 uAP, while the winning video systems reach 87.2 uAP on detection and 91.5 on localization. Frame-level description plus temporal alignment is not a refinement over single-vector matching — it is most of the result. In Mixpeek this maps onto the pipeline directly. Video extractors sample and embed at scene and frame granularity instead of producing one vector per file; every index entry carries a (video_id, timestamp) payload, which is what lets a retriever return the matching moment rather than the matching file; and retriever stages compose the cheap pass with the expensive one — fingerprint-style matching to catch exact and near-duplicate copies, then embedding search for semantic lookalikes. Honest scope note: Mixpeek is not a VSC23 submission and we do not claim these uAP numbers. The benchmark is the reason our video indexing is frame-level and timestamp-carrying by default, and it is the standard we read our own reverse video search behavior against.