Does Combining Two Embedding Models Improve Image Search? Measured on COCO and Flickr30k

Does combining two embedding models improve image search?

It can, by about two points of recall@1, when the two models come from different families and the search adds up both models' scores for each image. In our runs, SigLIP 2 So400m/14 at 384 px fused with Apple's DFN5B CLIP ViT-H/14 put the right photo first for 58.10% of COCO's 25,010 test captions. SigLIP 2 alone scored 55.81%. The best published text-to-image recall@1 among open models we could run is 56.0 (SigLIP 2 So400m/16 at 384 px), so the pair beat every single model on the bar. On Flickr30k the two-model gain at rank 1 was +0.66 points, inside the noise, and fusing four models added +1.58.

Two things decide whether a second model helps at all:

1. The fusion has to combine both models' evidence for the same image. If each model indexes the photos in its own collection and you fuse the two result lists by document id, every photo shows up twice and the top result never changes. With reciprocal rank fusion keyed on document id, rank 1 was identical to the single model on all 25,010 COCO captions. 2. Query preprocessing is worth more than a second model. Lowercasing text queries for SigLIP 2 added +3.71 recall@1 on COCO and +4.22 on Flickr30k, because the model was trained on lowercased text. Why SigLIP 2 search results are bad covers that fix and the padding one.

What we measured

Text-to-image retrieval on the standard test splits: every caption is a query, the index holds the test images, and recall@k is the share of captions whose own image appears in the top k results. COCO 5k test has 5,000 images and 25,010 captions. Flickr30k 1k test has 1,000 images and 5,000 captions. Every run went through Mixpeek's normal ingestion and retriever path in November 2026, on the open checkpoints from Hugging Face. Recall is in percent.



The four-model run fused SigLIP 2 So/14 and So/16, DFN5B H/14 and MetaCLIP H/14. Its COCO run did not fit in the time we had on the GPU machine.

Each gain below is paired: both retrievers answer the same captions, and the 95% interval comes from 2,000 bootstrap draws over those captions. An interval that crosses zero means the change might do nothing.



Fusion moves results in both directions. On COCO the two-model fusion ranked the right photo first on 1,438 captions where SigLIP 2 alone missed it, and missed it on 864 where SigLIP 2 alone had it right. The +2.30 is the net of those two counts. Expect the same on your data, and check the losses as well as the average.

A trap in the published numbers

The SigLIP 2 paper's Table 1 lists DFN5B H/14 at 63.1 on COCO. That figure averages the two retrieval directions. Apple's own eval_results.jsonl gives text-to-image recall@1 53.96 and image-to-text 72.26, and (53.96 + 72.26) / 2 is 63.1. Our run of DFN5B scored 53.89 text-to-image, in line with Apple's number. If a single model seems to beat everything on a leaderboard, check which direction the column reports.

Why rank fusion by document id cannot change the top result

When two models each index the same photos in their own collection, one photo has two document ids. Fuse the two result lists by document id and the photo appears twice, each copy carrying one model's evidence. Reciprocal rank fusion (RRF) scores a hit 1 / (k + rank) with k = 60, so both lists' first hits score 1/61 and tie, and the tie goes to whichever list was read first. The second model fills ranks 2 to 10 and never moves rank 1.

Fusing by the image instead adds both models' evidence into one score per photo. This script shows the difference on three photos:

from statistics import mean, pstdev

# Two models index the same photos, each in its own collection, so one photo
# has a different document id in each result list ("s:dog" and "d:dog").
siglip = [("s:beach", 0.31), ("s:dog", 0.29), ("s:park", 0.20)]
dfn = [("d:dog", 0.36), ("d:park", 0.33), ("d:beach", 0.22)]

def by_document(doc_id):
    return doc_id

def by_photo(doc_id):
    return doc_id.split(":")[1]

def rrf(result_lists, key, k=60):
    scores = {}
    for results in result_lists:
        for rank, (doc_id, _) in enumerate(results, start=1):
            scores[key(doc_id)] = scores.get(key(doc_id), 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])

def dbsf(result_lists, key):
    scores = {}
    for results in result_lists:
        values = [score for _, score in results]
        low = mean(values) - 3 * pstdev(values)
        high = mean(values) + 3 * pstdev(values)
        for doc_id, score in results:
            scores[key(doc_id)] = scores.get(key(doc_id), 0) + (score - low) / (high - low)
    return sorted(scores.items(), key=lambda kv: -kv[1])

for name, ranked in [
    ("RRF by document", rrf([siglip, dfn], by_document)),
    ("RRF by photo", rrf([siglip, dfn], by_photo)),
    ("DBSF by photo", dbsf([siglip, dfn], by_photo)),
]:
    print(f"{name:16} " + ", ".join(f"{doc} {score:.4f}" for doc, score in ranked[:3]))

It prints:

RRF by document  s:beach 0.0164, d:dog 0.0164, s:dog 0.0161
RRF by photo     dog 0.0325, beach 0.0323, park 0.0320
DBSF by photo    dog 1.2382, beach 0.9202, park 0.8416

By document, the beach photo wins a tie it only holds because SigLIP's list was read first. By photo, the dog wins because both models rank it high.

Rank fusion or score fusion?

RRF uses only each hit's rank, so a model that is barely sure of its first result counts as much as one that is very sure. That makes RRF robust when two systems score on unrelated scales, such as BM25 and cosine similarity, and it throws away the margin between results. Distribution-based score fusion (DBSF) keeps the margin: it rescales each model's scores against that query's own result list, mapping three standard deviations either side of the mean onto 0 to 1, and then adds the rescaled scores per image. Both fused retrievers in the tables above used DBSF keyed on the image. Hybrid search fusion goes deeper on normalization.

What does a second model cost?

Every image is embedded twice at ingestion, every query is embedded twice at search time, and the index stores two vectors per image. SigLIP 2 So400m vectors have 1,152 dimensions (4,608 bytes as float32) and DFN5B H/14 vectors have 1,024 (4,096 bytes), so the pair takes 8,704 bytes per image, 1.9 times SigLIP 2 alone. For 10 million images that is about 87 GB of raw vectors instead of 46 GB, before index overhead. Run the two searches in parallel and query latency is the slower of the two, plus a fusion step over the hits they return.

When to combine models, and when to keep one

Combine two models when:

Recall at rank 1 decides the product, as it does for "find this exact photo" search, and you can pay for two embeddings per image.

The models come from different training data. SigLIP 2 and DFN5B were trained on different web image sets, and the pair gained +2.30 on COCO.

You have already fixed query preprocessing for each model.

Keep one model when:

Your own evaluation puts the gain inside the noise, as Flickr30k did for the two-model pair.

Ingestion cost or index size is the binding constraint.

Your users mostly scroll a page of results. The pair's gain at recall@10 was +1.59 on COCO, smaller than at rank 1.

How to test fusion on your own images

1. Collect 200 to 2,000 queries with the image each should return. Real search logs beat synthetic captions. 2. Score each model alone with its own query preprocessing, at recall@1, @5 and @10. 3. Index both models over the same images and fuse per image, with RRF and with DBSF. 4. Compare on the same queries with a paired bootstrap, and count the queries won and lost. Keep the second model only if the interval at the k your product shows stays above zero.

Doing this in Mixpeek

Index the same bucket into two collections, one per model, and put both searches in one feature_search stage with fusion set to dbsf and fuse_on set to source_object. Each photo then comes back as one result whose score adds both models' evidence. With the default, fuse_on: document, a photo indexed by two models comes back as two results that never combine. Since November 2026 Mixpeek lowercases SigLIP and SigLIP 2 text queries itself, so the query template needs no lower filter.

{
  "stage_name": "feature_search",
  "stage_type": "filter",
  "config": {
    "stage_id": "feature_search",
    "parameters": {
      "searches": [
        {
          "feature_uri": "<your SigLIP 2 feature URI>",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" },
          "top_k": 100
        },
        {
          "feature_uri": "<your DFN5B feature URI>",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" },
          "top_k": 100
        }
      ],
      "fusion": "dbsf",
      "fuse_on": "source_object",
      "final_top_k": 25
    }
  }
}

The feature search docs list every fusion option. To compare embedding models before you pick two, see the best multimodal embedding models and the SigLIP 2 So400m model page.

Frequently Asked Questions

Does combining two embedding models improve image search?

It can. Fusing SigLIP 2 So400m/14 with DFN5B CLIP H/14 per image raised text-to-image recall@1 on COCO from 55.81% to 58.10%, a paired gain of +2.30 points (95% interval 1.93 to 2.69). On Flickr30k the same pair gained +0.66 at rank 1, inside the noise. The gain depends on fusing by image and on the models coming from different families.

Why does fusing two embedding models not change my top result?

You are probably fusing by document id while each model indexes the images in its own collection. Every image then appears twice in the fused list, once per model, and with reciprocal rank fusion both lists' first hits tie at 1/61. Fuse by the source image so each photo gets one score that adds both models' evidence.

Should I use RRF or DBSF to combine embedding models?

RRF combines ranks and ignores scores, which suits systems on unrelated scales such as BM25 and vectors. DBSF rescales each model's scores against the query's own result list and adds them, which keeps how confident each model is. For two image-text embedding models searching the same images, both fused results above used DBSF keyed on the image.

Is lowercasing queries or adding a second model the bigger gain?

Lowercasing, for SigLIP and SigLIP 2. It added +3.71 recall@1 on COCO and +4.22 on Flickr30k, more than the second model's +2.30 and +0.66, and it costs nothing. Fix each model's query preprocessing first, then measure whether a second model still helps.

How much does a second embedding model cost?

Twice the embedding compute at ingestion and at query time, and a second vector per image. SigLIP 2 So400m (1,152 dimensions) plus DFN5B H/14 (1,024 dimensions) take 8,704 bytes per image as float32, about 87 GB of raw vectors for 10 million images against 46 GB for SigLIP 2 alone.

Which open model has the best text-to-image recall?

Of the open checkpoints compared here, SigLIP 2 g/16 has the highest published COCO text-to-image recall@1 (56.1), followed by SigLIP 2 So400m/16 at 384 px (56.0), So400m/14 at 384 px (55.8) and DFN5B H/14 at 378 px (55.6). Fusing SigLIP 2 So400m/14 with DFN5B H/14 per image reached 58.10 in our runs, above all of them.

Sources

Tschannen et al., SigLIP 2 (arXiv 2502.14786), Table 1

Apple DFN5B-CLIP-ViT-H-14 on Hugging Face and its eval_results.jsonl

google/siglip2-so400m-patch14-384 on Hugging Face

Cormack, Clarke and Buettcher, Reciprocal Rank Fusion (SIGIR 2009)