Image Understanding: Vision Encoders & Multimodal Search

You've used text search. Now imagine searching images with text, or images with images.

Image understanding is the next evolution of semantic search. Instead of encoding text into vectors, we encode images into the same vector space. This enables powerful cross-modal retrieval: text queries returning images, image queries returning text, or image queries returning similar images.

From Text to Vision

If you've used a chatbot or Google, you've done text search: supply a text query, get text results. But what about:

• Text → Image: "red sneakers" returns product photos
• Image → Text: Upload a photo, get descriptions or captions
• Image → Image: "More like this" visual similarity

This is multimodal search, and it's powered by vision encoding models.

How Vision Transformers Work

Think of image understanding as text understanding, but for pixels.

Text Encoding (Review)

With text, we use models like BERT:

text = "red sneakers"
embedding = bert_model.encode(text)  # [0.23, -0.45, 0.78, ...]

Image Encoding

With images, we use vision transformers like CLIP or SigLIP:

image = load_image("sneakers.jpg")  # Base64 or pixel array
embedding = clip_model.encode(image)  # [0.21, -0.43, 0.79, ...]

The magic: both text and image embeddings live in the same vector space. Similar concepts, whether expressed as text or images, map to similar vectors.

Image Patches: Breaking Down the Visual

How does a vision transformer "see" an image?

Step 1: Divide into Patches

The image is split into small squares called patches. A typical configuration:

Each patch is a small window into part of the image.

Step 2: Embed Each Patch

Each patch is converted into a vector using a transformer:

patches = split_into_patches(image, patch_size=16)
patch_embeddings = [transformer(patch) for patch in patches]
# Result: 196 vectors, each 768 dimensions

Step 3: Mean Pooling

The patch embeddings are combined into a single image embedding:

# Average all patch embeddings into one vector
image_embedding = mean(patch_embeddings)
# Result: 1 vector, 768 dimensions

This is a simplified explanation: actual models use attention mechanisms and positional encodings, but the core idea holds, images become fixed-dimension vectors.

CLIP: The Foundation

CLIP (Contrastive Language-Image Pre-training) was published by OpenAI and changed everything.

How CLIP Works

CLIP is trained on 400 million image-text pairs from the internet. For each pair:

1. Encode the image → image embedding
2. Encode the text → text embedding
3. Push matching pairs closer together
4. Push non-matching pairs farther apart

After training, the model understands that "a photo of a dog" and an actual dog image should have similar embeddings.

CLIP Properties

Using CLIP

from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Encode image
image = load_image("red_sneakers.jpg")
image_embedding = model.get_image_features(processor(images=image, return_tensors="pt"))

# Encode text
text = "red sneakers"
text_embedding = model.get_text_features(processor(text=text, return_tensors="pt"))

# These can be compared directly!
similarity = cosine_similarity(image_embedding, text_embedding)

SigLIP: Google's Improvement

SigLIP (Sigmoid Loss for Language-Image Pre-training) is Google's refinement of CLIP.

Key Differences

Why SigLIP?

SigLIP's sigmoid loss is more compute-efficient and often achieves better zero-shot classification. In Mixpeek, we use SigLIP variants for image extraction by default.

# Mixpeek uses SigLIP under the hood
feature_uri = "mixpeek://image-extractor/v1/google-siglip-base"
# This maps to Google's SigLIP model with 768 dimensions

What Embedding Dimensions Capture

A 768-dimension embedding captures abstract visual concepts:

• Color: Red vs blue vs green
• Shape: Round vs rectangular vs irregular
• Texture: Smooth vs rough vs patterned
• Style: Modern vs vintage vs artistic
• Objects: Sneakers vs boots vs sandals
• Context: Indoor vs outdoor vs studio

The model learns these "concepts" during training. You can't point to dimension 147 and say "this is color": the representations are distributed across all dimensions.

Example: Red Sneakers

image = load_image("red_sneakers.jpg")
embedding = model.encode(image)

# This embedding encodes:
# - The redness (color concept)
# - The sneaker shape (object concept)
# - The style (aesthetic concept)
# - And hundreds more abstract features

When you search for "red sneakers" as text, the text encoder produces a similar embedding, enabling cross-modal matching.

Object Detection with YOLO

Embeddings capture the whole image. What if you need to find specific objects?

YOLO: You Only Look Once

YOLO is a family of models for real-time object detection. Unlike embedding models that produce vectors, YOLO produces bounding boxes with labels.

from ultralytics import YOLO

model = YOLO("yolov8n.pt")
results = model("street_scene.jpg")

# Returns:
# [
#   {"label": "car", "confidence": 0.95, "bbox": [100, 200, 300, 400]},
#   {"label": "person", "confidence": 0.89, "bbox": [350, 150, 450, 500]},
#   {"label": "dog", "confidence": 0.87, "bbox": [500, 300, 600, 450]}
# ]

YOLO Capabilities

When to Use YOLO vs Embeddings

Cross-Modal Search in Action

Let's build a real multimodal retrieval system.

Step 1: Index Images

from mixpeek import Mixpeek

client = Mixpeek(api_key="your_key")

# Create collection with image extractor
collection = client.collections.create(
    name="gallery",
    feature_extractors=[
        {
            "uri": "mixpeek://image-extractor/v1/google-siglip-base",
            "apply_to": "image"
        }
    ]
)

# Index images from S3 bucket
client.buckets.connect(
    collection_id=collection.id,
    bucket_uri="s3://my-bucket/images/"
)

Each image is:

1. Downloaded from S3
2. Converted to patches
3. Embedded using SigLIP
4. Indexed into a vector database

Step 2: Build a retriever over the gallery

Search runs through a retriever: you create it once, then execute it with different inputs. Its feature_search stage matches your query against the embeddings the collection was indexed with (the SigLIP image embeddings from Step 1). The input_schema declares what you can pass at query time.

# Create the retriever once
retriever = client.retrievers.create(
    name="gallery-search",
    input_schema={
        "query": {"type": "text", "required": False},
        "query_image_url": {"type": "image", "required": False},
        "document": {"type": "document_reference", "required": False},
    },
    stages=[{"type": "feature_search", "top_k": 10}],
)

Step 3: Text-to-Image Search

# The text is encoded with the SAME model the gallery was indexed with,
# then matched against the image embeddings.
results = client.retrievers.execute(
    retriever_id=retriever.id,
    inputs={"query": "dogs outside"},
)

for doc in results["documents"]:
    print(f"Score: {doc['score']}, Image: {doc['file_url']}")

Step 4: Image-to-Image Search

# Search using a reference image
results = client.retrievers.execute(
    retriever_id=retriever.id,
    inputs={"query_image_url": "https://example.com/reference_dog.jpg"},
)

# Returns visually similar images

Step 5: Combined Text + Image Search

Here's where it gets powerful. Pass both a text and an image input:

results = client.retrievers.execute(
    retriever_id=retriever.id,
    inputs={
        "query": "dogs outside",
        "query_image_url": "https://example.com/my_australian_shepherd.jpg",
    },
)

Under the hood, the text and image are each encoded and combined into one query, so results match both the concept "dogs outside" and the look of your Australian Shepherd.

Document Reference Search

You can also use an existing document as the query: "find more like this." Pass a document reference (its collection plus id), not a bare string. The retriever reuses that document's pre-computed embedding rather than re-encoding anything.

results = client.retrievers.execute(
    retriever_id=retriever.id,
    inputs={"document": {"collection_id": "gallery", "document_id": "doc_12345"}},
)

This is document-to-document similarity: it matches on doc_12345's stored features directly, with no re-inference.

Feature URIs: Match the Index, Not the Query

Image embeddings are versioned by Feature URI:

Different Models, Different Dimensions

Why This Matters

The Feature URI is chosen when you index the collection, not when you query. A retriever's feature_search searches whatever embeddings the collection holds, so a query is automatically compared in the same embedding space the data was indexed in. To search the same images with a different model, you re-index the collection with that model's extractor: you cannot mix embedding spaces at query time.

Real-World Demo: National Gallery

The video demonstrates a live retriever on 118,000 images from the National Gallery of Art.

The Setup

• Collection: 118,000 portrait images
• Feature URI: mixpeek://image-extractor/v1/google-siglip-base
• Dimensions: 768
• Live demo: mxp.co/r/nga

Text Search

Image Search

Upload a reference image → find visually similar portraits.

Combined Search

The retriever combines text and image embeddings using mean pooling: no hybrid search fusion needed because both modalities share the same Feature URI.

Architecture Summary

Key Takeaways

1. Image understanding = encoding images into the same vector space as text
2. Vision transformers (CLIP, SigLIP) convert images into patches → embeddings
3. 768 dimensions capture color, shape, texture, style, objects, and more
4. YOLO detects specific objects; embeddings enable semantic similarity
5. Cross-modal search: text→image, image→image, image→text all work
6. Mean pooling combines text + image queries into a single embedding
7. Feature URIs ensure query and index use the same embedding model
8. Document reference queries reuse existing embeddings

What's Next?

With image understanding covered, explore:

• Video Understanding: Scene detection, temporal analysis, shot boundaries
• Audio Understanding: Speech, music, sound effects as embeddings
• Multimodal RAG: Combining retrieval with generative AI

Resources

• Mixpeek Image Extractor Docs
• CLIP Paper (OpenAI)
• SigLIP Paper (Google)
• YOLO Documentation
• Live Demo: National Gallery Retriever

Try It Yourself

1. Index 1,000 images from your photo library using the image extractor
2. Search with text queries: do the results match your expectations?
3. Upload a reference image and find "more like this"
4. Combine text + image queries to narrow results

Discussion Questions

• When would you choose YOLO over embedding-based search?
• How does patch size affect embedding quality?
• What happens when your query concept wasn't in the training data?
• How would you handle images with multiple distinct objects?