How do I stop duplicate videos showing up in my search results?
First find out which kind of duplicate you have, because each has a different fix. Exact copies of the same file are removed at ingest by a content hash. Near-copies (re-encoded, resized, trimmed or watermarked versions) are found with a video fingerprint or embedding and linked to one original. And several results from the same long video are usually not duplicates at all but separate clips of it, which you collapse in the results by grouping on the source video. Most libraries have all three.
If you are seeing duplicates on YouTube or another site you do not run, the site decides what it shows; this page is about a library or search system you operate.
Where do duplicate videos come from?
| Kind | Example | How to detect it | Where to fix it |
| Exact copy | The same file uploaded twice, or synced from two folders | Identical content hash (for example SHA-256) | At ingest: skip it or link it to the first copy |
| Near-copy | A re-export at a lower bitrate, a resized social cut, a watermarked version | Video fingerprint or keyframe embeddings, compared with a threshold | At ingest or in a cleanup job: link it to the original and choose which to show |
| Trimmed or edited version | A 15-second cut taken from a 60-second ad | Matching segments: part of one video matches part of another | In a cleanup job, recording which is derived from which |
| Many clips from one video | A search matching five scenes of the same hour-long recording | Every result shares the same source video id | In the results: group by source video and show the best clip |
How do I remove exact copies?
Hash every file's bytes when it arrives and keep one record per hash. A second upload with the same hash is the same video whatever its file name, so it can be skipped or pointed at the first record. This is cheap, catches every byte-identical copy, and should run before anything expensive such as transcription or embedding, so duplicates do not cost processing either.
How do I find near-copies that are not byte-identical?
Compare what the videos look like. Two common methods:
Link each near-copy to one original rather than deleting it, and decide per library which version to show: usually the highest quality or the earliest upload.
Why does the same video appear several times in one search?
Usually because the index stores clips, not whole videos. Long videos are split into scenes or segments so a search can return the moment; a query that matches several moments of one video returns several clips of it. The fix is in the results: group results by the source video, keep the best-scoring clip from each, and offer "more from this video" for the rest. Deleting clips would lose the timestamps the search exists to find.
How do I do this with Mixpeek?
Mixpeek fingerprints every object with a SHA-256 content hash on the way in, so exact copies are recognised at ingest and not processed twice. Videos are indexed as segments with their start and end times. In a retriever, the group-by stage collapses results to one per source video, and the deduplicate stage removes near-identical results by field or by content similarity. The videos stay in your own object storage.
Related: the best video deduplication tools, how to find duplicate photos, including edited copies, how do I find out if someone reposted my video and why video search returns the wrong scene.
Frequently Asked Questions
Why do I get the same video more than once in my search results?
Either the library holds several copies of it (re-uploads, re-encodes, trimmed versions), or the search returns clips and several clips of one video matched. Hash files at ingest for the first case, link near-copies with a fingerprint or embedding, and group results by source video for the second.
How do I detect duplicate videos that were re-encoded or resized?
Compare the pictures, not the files. A video fingerprint built from perceptual hashes of sampled frames matches re-encoded and resized copies; frame or clip embeddings also match crops, colour changes and overlays.
Should I delete duplicate videos?
Delete exact copies once you have kept one; link near-copies and edits to their original instead, because a trimmed or watermarked version is often used on purpose. Keep a record of which version is the original.
How do I show one result per video instead of every matching clip?
Group the results by the source video's id and keep the highest-scoring clip from each group, with the others available under that result. The timestamps of the other matching clips stay useful.