Visual search that handles negation, attribute bleed and unseen classes without retraining

Why does visual search fail on "not a dog" and "white dog"?

Image embeddings from CLIP-style models encode a query as one vector, and that vector carries the words it contains more than how they relate. So "a photo that is not a dog" lands near dog photos, and "white dog" returns a few white dogs, then white cars, white walls and other white things. The fix that works without retraining is to stop asking the vector to carry the whole query: let the embedding recall candidates, check the attributes and negations on the candidates, and learn from the queries that still fail. This guide shows that pipeline on Mixpeek and where each piece stands today.

What the research says

Two results explain most of what search teams see in production.

Negation. NegBench (Alhamoud et al., 2025) tested vision-language models on retrieval with negated queries across 79,000 examples in image, video and medical data. Modern models often performed at chance. Fine-tuning CLIP on millions of synthetic negated captions raised recall on negated queries by about 10%.

Attribute binding. The ARO benchmark (Yuksekgonul et al., ICLR 2023) found that vision-language models blunder when linking objects to their attributes and largely ignore word order. The authors show that a model can score well on standard retrieval benchmarks without using composition at all, which is why the weakness survives training.

That is why flipping the sign of the "dog" component does not give you "not a dog": the embedding space was never trained to put negated captions anywhere meaningful, so subtracting a direction moves the query somewhere arbitrary.

The failure modes, and what fixes each

The first three need no new model. The last one is where learning from feedback pays off.

Step 1: recall with the embedding, then check what the vector cannot

Keep the embedding for what it does well, finding visually similar candidates fast, and move negation and attribute checks to a later stage that reads each candidate. On Mixpeek that is a retriever with three stages:

{
  "retriever_name": "visual-search-with-checks",
  "collection_identifiers": ["product_images"],
  "stages": [
    {
      "stage_name": "recall",
      "config": {
        "stage_id": "feature_search",
        "parameters": {
          "searches": [{
            "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
            "query": { "input_mode": "text", "value": "{{INPUT.positive}}" },
            "top_k": 100
          }]
        }
      }
    },
    {
      "stage_name": "check",
      "config": {
        "stage_id": "llm_filter",
        "parameters": {
          "criteria": "Keep the image only if it matches: {{INPUT.query}}. Treat words like no, not and without as exclusions, and check colors and other attributes on the main subject.",
          "include_media": true
        }
      }
    },
    {
      "stage_name": "diversify",
      "config": { "stage_id": "mmr", "parameters": { "lambda": 0.7, "top_k": 24 } }
    }
  ]
}

Recall runs feature search on the positive part of the query ("beach photos"), so the negation never reaches the vector.

Check runs the LLM filter with include_media: true, which sends each candidate's image to a vision model with the full query, so "no people" and "white" are judged on what the image shows.

Diversify applies MMR, which stops ten near-identical shots from filling the first page.

Splitting the query into its positive part and its exclusions can happen in your application, or with the query expand stage, which generates query variations with a language model and fuses their results. Where your images carry text (captions, product titles), a rerank stage with a cross-encoder can reorder the survivors.

Budget the check. A model call per candidate is the expensive part, so run the check on the top 50 to 100 candidates only. The vector search stays the fast path, and the check touches only what is about to be shown.

Step 2: onboard unseen classes without retraining

A new logo or product does not need a new model if you can show the system a few examples of it. Index a handful of labeled exemplars per class as documents in a collection, with the class name in their metadata, and match incoming images against them. The class lives in the index, so adding one is an upload, and removing one is a delete.

Two details keep this reliable at large scale:

Pin the embedding version. Mixpeek addresses every embedding by a feature URI such as mixpeek://multimodal_extractor@v1/..., and a retriever pinned to that URI keeps searching the space the exemplars were indexed in. When you do change models, model migration moves the index to the new space as a separate step.

Set thresholds per class. A logo with simple geometry matches lookalikes more readily than a detailed one. A similarity threshold per class, stored with the exemplars, keeps confidence honest across a large catalog.

When a class needs a model of its own, Mixpeek runs your own weights inside a custom extractor on Ray, so a fine-tuned embedding model can sit next to the stock one.

Step 3: collect the signals while you serve

A system that improves needs to know what happened after each search. Mixpeek stores both kinds of signal as collections in your namespace, which you can filter, cluster and export like any other (system collections):

Weak signals. POST /v1/retrievers/interactions records clicks, dwells, skips and other interactions with the result's position, into mxp_retriever_interactions.

Strong signals. Human labels on results land in mxp_document_annotations, keyed to the document and the annotator.

The queries themselves. Every query is stored with its embedding in mxp_retriever_queries.

Step 4: find what is failing

With queries, results and outcomes in collections, finding the failure modes is a clustering job. Run a cluster over query embeddings and look at the groups with high skip rates, low click-through or zero results. Those groups name the intents your embedding space gets wrong ("X without Y", one color on one object class) far faster than reading logs. Clustering query and result pairs, or the interactions themselves, separates a bad query type from a bad document.

Step 5: learn a correction from the failing queries

Once you have failing queries and the results that should have come back, you can correct the query side without touching the index. The idea: learn a transform, often a single matrix, that maps a query embedding to a point closer to the documents users wanted and further from the ones they skipped. Documents stay where they are, so there is no re-indexing and no full fine-tune.

Chroma's 2024 technical report on embedding adapters measured this on text retrieval: a linear transform applied only to query embeddings, trained on as few as about 1,500 labeled query and document pairs, improved retrieval across many domains, by up to 70% NDCG on one benchmark.

For visual search the same recipe applies:

1. Take the failing query clusters from step 4. 2. Pair each query with clicked or annotated positives and skipped negatives. 3. Fit the transform with a contrastive loss. 4. Apply it to new queries in the matching cluster at search time. 5. Keep the version of the transform next to the feature URI, so you can say which correction produced which result.

Where this stands on Mixpeek: the signals, clustering and the stages in steps 1 to 4 ship today. A retriever stage that learns and applies a query transform is not in the product yet. Today the transform runs as your own code over exported collections.

Running it in your own AWS account

Teams with large image indexes usually cannot send them anywhere. Customer-hosted Mixpeek runs in your cloud account: the vector store on your disks and object storage, Ray on your nodes, with Mixpeek operating its software inside a namespace you grant. The deployment models page lists who owns what in each setup. The interactions, annotations and clusters above stay in your account with the images.

FAQ

How do I handle negation in vector search?

Search on the positive part of the query, then check the negation on the candidates with a model that reads each image, such as an LLM filter stage with the image attached. Vision-language embeddings often perform at chance on negated queries (NegBench, 2025), so the vector alone will not exclude what you asked to exclude.

Why does "white dog" return white objects that are not dogs?

CLIP-style embeddings bind attributes to objects weakly, so the query vector partly means "white" and partly "dog" (ARO benchmark, ICLR 2023). The first results are usually right; further down, things that are only white score well. Checking the attribute on the main subject of each candidate and diversifying with MMR keeps those out of the visible results.

How do I add new classes, like new logos, without retraining the model?

Index a few labeled exemplars of each new class and match incoming images against them, with a similarity threshold per class. Pin the retriever to the embedding version the exemplars were indexed with, so new and old classes live in the same space.

Can image search learn from click feedback?

Yes, if clicks, skips and labels are stored with the query and the result position. Cluster the queries to find the intents that fail, then use those queries and their positives and negatives to learn a correction or tune ranking. On Mixpeek, interactions and annotations are collections you can cluster and export today; a stage that learns a query correction is an approach, not a shipped feature.

Can a self-improving visual search system run in my own AWS account?

Yes. Customer-hosted Mixpeek runs the vector store, Ray and the retrieval stages inside your cloud account, so images, embeddings and feedback never leave it.

Sources

Alhamoud et al., "Vision-Language Models Do Not Understand Negation" (NegBench), 2025: https://arxiv.org/abs/2501.09425

Yuksekgonul et al., "When and why vision-language models behave like bags-of-words, and what to do about it?", ICLR 2023: https://arxiv.org/abs/2210.01936

Sanjeev and Troynikov, "Embedding Adapters", Chroma technical report, May 2024: https://research.trychroma.com/embedding-adapters

Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021: https://arxiv.org/abs/2103.00020