What is Mixpeek?

[Back to Blog](/blog)

[](https://twitter.com/intent/tweet?text=Multimodal%20Image%20Search%20with%20SigLIP%20and%20RRF&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fvisual-search-rrf)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fvisual-search-rrf&title=Multimodal%20Image%20Search%20with%20SigLIP%20and%20RRF)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fvisual-search-rrf&title=Multimodal%20Image%20Search%20with%20SigLIP%20and%20RRF)

This was originally in our newsletter.

Get Updates

Dec 31, 2025•3 min read

# Multimodal Image Search with SigLIP and RRF

Search 120K pieces of art by text, image, or both. How we built a multimodal retriever with SigLIP and reciprocal rank fusion.

![Multimodal Image Search with SigLIP and RRF](https://blog.mixpeek.com/content/images/2025/12/npg-header-1.jpg)

Industry

TABLE OF CONTENTS

We built a visual search engine for the National Gallery of Art's public collection. Text search, image search, and hybrid queries with reciprocal rank fusion.

[**Demo**](https://nga.mxp.co/?ref=blog.mixpeek.com) | [**Code**](https://github.com/mixpeek/showcase/tree/main/gallery?ref=blog.mixpeek.com) **|** [**Video**](https://mixpeek.com/videos/exploratory-multimodal-retriever?ref=blog.mixpeek.com)

* * *

## The Stack

*   **Feature Extractor**: [Google SigLIP (768-dim vectors)](https://mixpeek.com/docs/processing/extractors/image?ref=blog.mixpeek.com)
*   **Stack**: FastAPI -> Celery -> Ray -> Mixpeek Vector Store
*   **Resources:** 2× NVIDIA L4, 32GB on GCP via Anyscale

💡

~2 hours for 120K images (~60ms/image)

## Why SigLIP over CLIP

CLIP uses softmax loss—it optimizes for _relative_ ranking within a batch. SigLIP uses sigmoid loss, treating each image-text pair as independent binary classification.

_Practical difference:_ SigLIP embeddings live in a global semantic space. Similarity scores are consistent whether you're comparing 10 documents or 10 million. Better for retrieval at scale.

The base model, `siglip-base-patch16-224`, hits ~84% zero-shot on ImageNet. Good enough out of the box, no fine-tuning needed for general visual similarity.

## Pipeline

![](https://blog.mixpeek.com/content/images/2025/12/image.png)

The collection config:

```json
{
  "feature_extractor": {
    "feature_extractor_name": "image_extractor",
    "version": "v1",
    "input_mappings": { "image": "image" },
    "parameters": {
      "model": "siglip-base-patch16-224",
      "generate_thumbnail": true
    }
  }
}
```

Thumbnails generated and pushed to Cloudfront. The entire batch (120k images) runs in a dedicated Ray job as a feature extractor to fully saturate our GPUs/CPUs, orchestrated by Anyscale in GCP scaling up/down accordingly.

💡

[Feature extractors](https://mixpeek.com/extractors?ref=blog.mixpeek.com) are fully-managed indexing pipelines composed of models, workflows and code to deliver SOTA retrieval for any filetype at terabyte scale.

The output is then stored in a collection as documents, ready for retrieval.

## The Retriever

When we create the retriever, we provide inputs to the stages using standard Jinja templating. So we now have three query types in one `feature_search` stage:

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_mid&utm_campaign=mvs_onramp)[Talk to us](/demo)

```json
{
  "stage_id": "feature_search",
  "parameters": {
    "searches": [
      {
        "feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
        "on_empty": "skip",
        "query": {
          "input_mode": "text",
          "value": "{{INPUT.text}}"
        },
        "top_k": 250
      },
      {
        "feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
        "on_empty": "skip",
        "query": {
          "input_mode": "content",
          "value": "{{INPUT.image}}"
        },
        "top_k": 250
      },
      {
        "feature_uri": "mixpeek://image_extractor@v1/google_siglip_base_v1",
        "on_empty": "skip",
        "query": {
          "input_mode": "document",
          "document_ref": {
            "collection_id": "{{INPUT.doc_ref.collection_id}}",
            "document_id": "{{INPUT.doc_ref.document_id}}"
          }
        },
        "top_k": 250
      }
    ],
    "final_top_k": 500,
    "fusion": "rrf"
  }
}
```

*   **Text**: "portrait of a scientist" → text encoder → kNN
*   **Image**: upload reference → vision encoder → kNN
*   **Document reference**: look up stored embedding → kNN (the "find similar" button)

`feature_uri` enables us to map the query to the index, embedding model, extractor, and version. Then feature search calls a hot Ray Serve node with fractional GPU availability.

💡

Learn more about [the feature search stage](https://mixpeek.com/docs/retrieval/stages/feature-search?ref=blog.mixpeek.com)

`skip_if_empty: true` means you pass whatever inputs you have. One query type? Runs that. Multiple? Fuses with RRF using default weights.

> We refer to this architecture as an **Exploratory Multimodal Retriever**: a single retrieval pipeline that accepts optional text, image, or document-reference inputs and produces a navigable similarity space.

## RRF for Hybrid Search

Reciprocal Rank Fusion merges ranked lists without caring about raw scores:

```
score(d) = Σ 1/(k + rank(d))
```

Why this matters: text-to-image similarity might cluster in \[0.2, 0.4\] while image-to-image clusters in \[0.6, 0.9\]. Score-based fusion would be biased. RRF normalizes by rank.

Learn more about RRF in our [hybrid search university module.](https://mixpeek.com/university/module/hybrid-search?ref=blog.mixpeek.com)

The killer query: pass a `document_id` (reference portrait) + `query_text` ("but wearing blue"). RRF combines structural similarity with the color constraint.

## Execution

Since this is a named retriever, when you call it you only provide the inputs and the mapping is done on execution time.

```rust
curl -X POST ".../retrievers/<RETRIEVER_ID>/execute" \
  -d '{
    "inputs": {
      "text": "dog outside"
    }
  }'
```

Response includes per-stage timing so you can see where latency lives.

```json
{
  "stage_name": "feature_search",
  "num_features": 3,
  "fusion_strategy": "rrf",
  "total_results": 250,
  "duration_ms": 899.33,
  "cache_hit": false
}
```

[https://mixpeek.com/docs/retrieval/retrievers](https://mixpeek.com/docs/retrieval/retrievers?ref=blog.mixpeek.com)  
  
Mixpeek caches retriever results by hashing normalized inputs and pipeline configuration, reusing full or stage-level outputs on repeat queries to cut latency and inference cost while respecting TTLs and invalidation rules. [About caching](https://mixpeek.com/docs/resources/best-practices?ref=blog.mixpeek.com)

## Numbers

Metric

Value

Images indexed

120,000+

Processing time

~2 hours

Embedding dimensions

768

Vector data size

~350MB

Query latency

<800ms p95

* * *

Same pattern works for product catalogs, media DAMs, real estate photos, medical imaging. Swap the data source, keep the architecture!

*   **Code**: [https://github.com/mixpeek/showcase/tree/main/gallery](https://github.com/mixpeek/showcase/tree/main/gallery?ref=blog.mixpeek.com)
*   **Demo**: [https://mxp.co/r/nga](https://mxp.co/r/nga?ref=blog.mixpeek.com)
*   **Video**: [https://mixpeek.com/videos/exploratory-multimodal-retriever](https://mixpeek.com/videos/exploratory-multimodal-retriever?ref=blog.mixpeek.com)

[ES

Ethan Steininger

Former lead of MongoDB's Search Team, Ethan noticed the most common problem customers faced was building indexing and search infrastructure on their S3 buckets. Mixpeek was born.



](/blog/authors/ethan)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_end&utm_campaign=mvs_onramp)[Talk to us](/demo)

## Continue Learning

[

![AI Video Analysis for Sports: Build Automated Highlight Reels, Archive Search, and Performance Analytics](https://blog.mixpeek.com/content/images/2026/03/sports-blog-hero.jpg)

### AI Video Analysis for Sports: Build Automated Highlight Reels, Archive Search, and Performance Analytics

Sports broadcasters cut 4-8 hour editing sessions to 15 minutes using AI video analysis. Learn how to build automated highlight detection, archive search, and performance analytics pipelines for any sport.

Read More





](/blog/ai-video-analysis-sports)[

![IAB Contextual Classifier: Taxonomies for Videos and Images](https://blog.mixpeek.com/content/images/2026/02/ChatGPT-Image-Feb-15--2026--12_50_33-PM.png)

### IAB Contextual Classifier: Taxonomies for Videos and Images

Classify text, images, and video into IAB Content Taxonomy 3.1 categories using multimodal AI. Learn how it works under the hood and how to extend it for your contextual targeting needs.

Read More





](/blog/iab-contextual-classifier-multimodal-ai)

### Author

ES

Ethan Steininger

![Mixpeek Icon](/brand/logo-dark-icon.webp)

#### Multimodal Search, On Demand

Index, retrieve, and reason over videos, images, audio, and docs, all from a single API.

[Start Building](/start)

### Share

[](https://twitter.com/intent/tweet?text=Multimodal%20Image%20Search%20with%20SigLIP%20and%20RRF&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fvisual-search-rrf)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fvisual-search-rrf&title=Multimodal%20Image%20Search%20with%20SigLIP%20and%20RRF)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fvisual-search-rrf&title=Multimodal%20Image%20Search%20with%20SigLIP%20and%20RRF)