The Short Answer
Your embedding model is doing exactly what it was trained to do. Models like CLIP and SigLIP learn to put visually and semantically similar things close together, so a red running shoe from a different brand lands very near the red running shoe you photographed. Both are "a red running shoe". That is the concept the model encodes.
Identity is a different question from similarity. "Find me this shoe" and "find me shoes like this" have the same query image and different correct answers, and a single similarity score cannot tell them apart. When you ask for the exact item and get the lookalike, the ranking is not broken. It is answering the other question.
To get identity you add a signal that only the same object can satisfy: geometric verification against the query image, a perceptual hash for re-encoded copies, or a hard filter on something written on the product itself.
Why the lookalike scores so well
Embedding models are trained with contrastive objectives. The model sees pairs that should be close and pairs that should be far apart, and it learns a space where distance means semantic resemblance. Colour, shape, texture, category and pose all push an image toward its neighbourhood.
Nothing in that objective rewards distinguishing two objects of the same type. A catalogue of one thousand black office chairs occupies a very small region of the space, and within that region the distances are dominated by lighting, crop and background rather than by which chair it is. Embedding space geometry goes into why cosine distance behaves this way and where intuition about it breaks.
There is a practical consequence people usually discover late. The tighter your catalogue, the worse this gets. A general image search over holiday photos rarely hits it. A product catalogue where half the items are variations of each other hits it on nearly every query.
Which question are you actually asking
It is worth being precise, because the fix differs.
Same object, different photo. You photographed a chair in a showroom and want that exact model from the catalogue. Lighting, angle and background all differ. This is instance retrieval.
Same file, altered. Someone re-encoded, cropped, resized or watermarked your image and you want to find those copies. This is near-duplicate detection and it is the easiest of the three.
Same kind of thing. You want visually similar products to recommend. Plain embedding similarity is already the right tool and you have no problem to solve.
Teams often want the first and build the third, because the third is what a vector search returns by default.
What to add, in increasing order of effort
A perceptual hash, for the altered-copy case. Hashes like pHash and dHash survive re-encoding and mild edits, and a hash match is a near-certain identity signal. Cheap to compute, cheap to store, and it answers a question embeddings answer badly. Perceptual image hashing covers what survives which transformation.
Geometric verification, for the same-object case. Retrieve candidates with the embedding, then verify each one by matching local keypoints between the query and the candidate and checking that the matches agree on a single consistent transform. Two photos of the same chair produce many matches that fit one geometry. A different chair of the same design produces scattered matches that fit nothing. This is the technique that actually separates identity from resemblance, and instance-level visual matching covers how the verification step works.
Read what is written on the object. Serial numbers, SKUs, model names and labels are identity, printed on the thing. Run text detection over the candidate images and match the string. This turns a visual problem into an exact-match problem, which is a solved one. Lexical matching on codes has its own failure mode, covered in why search cannot find exact part numbers.
Filter before you rank. If you know the category, brand or colourway, a hard constraint removes most of the lookalikes before scoring. Filtered vector search covers doing this without wrecking recall.
Why a higher similarity threshold does not fix it
The obvious reach is to raise the cutoff until lookalikes drop out. It fails in a specific way that is worth understanding.
Cosine scores are not calibrated. The absolute number depends on the model, the domain and the query image, so a threshold tuned on one catalogue does not transfer, and it usually does not even hold across queries in the same catalogue. Raise it far enough to exclude the lookalike and you start excluding genuine matches photographed at a worse angle. Calibrating similarity scores covers what the number actually means.
Retrieve generously and verify precisely. A cheap embedding search over the whole catalogue followed by an expensive check on the top candidates is the standard shape, and it is the same two-stage pattern as cross-encoder reranking with a geometric verifier in place of the reranker.
How to tell which problem you have
Take twenty queries where you know the right answer and look at where the correct item ranks.
If it is usually in the top fifty but rarely first, retrieval is fine and you need a verification stage. This is the common case and the cheapest to fix.
If it is often absent from the top few hundred, the embedding is not separating your domain at all, and a verification stage has nothing to work with. Look at domain-specific embeddings or an added text signal instead.
If it ranks first for clean catalogue photos and collapses for user-submitted ones, the gap is photometric rather than semantic. Crop to the object before embedding and the gap usually narrows.
Doing this on Mixpeek
Mixpeek runs the two stages in one retriever. An image extractor embeds the catalogue for candidate generation, and later stages apply filters and verification against the query image, so the ranking the caller receives has already been checked for identity rather than resemblance. Detected text on the product is indexed alongside the vector, which makes the SKU path available as a filter on the same query.
Compare the available vision and embedding models on models, read the storage side on MVS, and see per-extraction and per-query costs on pricing.
Frequently Asked Questions
Would a bigger or newer embedding model fix this on its own?
It raises the floor and does not change the shape. A stronger model separates categories better and still places two objects of the same type close together, because that is the training objective. The gain shows up as the correct item moving from rank forty to rank eight, which matters if you have a verification stage behind it and not much if you do not.
Is fine-tuning on my catalogue the answer?
It helps when your domain has distinctions a general model never learned, like subtle differences between industrial parts. It is a real cost in labelled pairs and maintenance, so try retrieve-then-verify first, since that often closes the gap without training anything.
Why does cropping the query image help so much?
An embedding summarises the whole frame. A product photographed on a desk encodes the desk, the lighting and the hand holding it, and all of that competes with the product for room in the vector. Detect the object, crop to it, embed the crop.
Can I just use an LLM to look at the top results and pick the right one?
You can, and it works reasonably well as the verification stage. Budget for it: it is a model call per query against several candidate images, which is far more expensive per query than geometric verification and easier to set up.
Does this apply to video too?
Yes, and more so. A frame embedding has the same similarity bias, and video adds near-identical adjacent frames, so a single object can occupy many positions in the result list. The verification step is the same; the candidate pool is larger.