Semantic Search Fundamentals

Keywords match words. Semantic search matches meaning.

Traditional keyword search looks for exact word matches: "heart attack symptoms" only finds documents containing those exact terms. Semantic search understands that "chest pain" and "cardiac arrest" are related concepts.

From Keywords to Meaning

The Keyword Problem

Keyword search relies on exact term matching:

This fails in two critical ways:

• No synonym understanding: Different words for the same concept don't match
• No domain awareness: Healthcare, legal, and technical terminology relationships are invisible

The Semantic Solution

Semantic search captures the concepts behind the words:

The prerequisite? An encoder model that understands these relationships.

Vector Similarity Algorithms

Once we encode content into vectors, we need ways to measure similarity. Two algorithms dominate:

Cosine Similarity

Measures the angle between two vectors: are they pointing in the same direction?

# King and Queen in vector space
king = [0.5, 0.7]
queen = [0.3, 0.9]

# Cosine similarity measures directional alignment
# Close to 1.0 = similar meaning
similarity = cosine_similarity(king, queen)  # High similarity

Cosine similarity ignores magnitude and focuses purely on direction. This makes it ideal for comparing semantic meaning regardless of document length.

Dot Product

Combines direction with vector strength: how confident is the signal?

# Dot product includes magnitude
similarity = dot_product(king, queen)

# Useful when confidence matters
# Stronger vectors = more certain embeddings

When to use which:

• Cosine similarity: Most semantic search (default choice)
• Dot product: When embedding confidence matters
• Euclidean distance: Less common, measures straight-line distance

How Semantic Search Works

The architecture behind every RAG system:

1. Encode the Query

When a user searches "heart attack symptoms":

query = "heart attack symptoms"
query_vector = embedding_model.encode(query)
# Result: [0.23, -0.45, 0.78, ...] (1408 dimensions)

2. Search the Index

Compare against pre-indexed documents using HNSW (Hierarchical Navigable Small World graphs):

# Documents already encoded and indexed
index = {
    "chest pain": [0.22, -0.44, 0.79, ...],
    "cardiac arrest": [0.21, -0.43, 0.77, ...],
    "headache relief": [-0.15, 0.32, 0.11, ...]  # Not similar
}

# Find nearest neighbors
results = hnsw_search(query_vector, index, k=5)

3. Return Ranked Results

Documents ranked by similarity score: highest relevance first.

Intent-Based Retrieval

Semantic search captures user intent, not just keywords:

The embedding model learns these intent mappings from training data.

Building Feature Search in Mixpeek

Feature search is the semantic search stage in Mixpeek's retrieval pipeline.

Creating a Retriever

# Configure the retrieval pipeline
retriever = mixpeek.retrievers.create(
    name="semantic-search",
    collection="scene-segments",
    stages=[
        {
            "type": "feature_search",
            "feature_uri": "multimodal-extractor-v1",
            "input_mode": "text",
            "limit": 5
        }
    ]
)

Multimodal Queries

Because the underlying embeddings are multimodal, you can search with:

• Text: Find videos matching a description
• Image: Find visually similar content
• Video: Find similar video segments

# Text-based search
results = retriever.search(query={"text": "dog playing"})

# Image-based search
results = retriever.search(query={"image": "reference.jpg"})

Chaining Stages

Feature search is just the first stage. Chain it with:

stages = [
    {
        "type": "feature_search",
        "limit": 20  # Cast a wide net
    },
    {
        "type": "llm_filter",
        "prompt": "Keep only documents related to dogs"
    }
]

The LLM filter refines results from feature search: semantic retrieval followed by intelligent filtering.

Key Takeaways

1. Keyword search matches words; semantic search matches meaning
2. Cosine similarity measures directional alignment between vectors
3. Encoder models are the prerequisite: they understand domain relationships
4. HNSW indexes enable fast approximate nearest neighbor search
5. Feature search is Mixpeek's semantic search stage: chain it with other stages for sophisticated retrieval

What's Next?

With semantic search fundamentals covered, explore:

• Retrieval Ranking Strategies: Combining multiple signals
• Reciprocal Rank Fusion: Merging results from different search methods
• Clustering: Visualizing semantic relationships in your data

Resources

• Mixpeek Feature Search Docs
• Understanding HNSW
• Multimodal Embeddings Guide

Discussion Questions

• When would keyword search outperform semantic search?
• How do you choose between cosine similarity and dot product?
• What happens when your encoder model doesn't understand your domain?