Hybrid Search: Best of Both Worlds

Keyword search has high precision, low recall. Semantic search has low precision, high recall. Hybrid search combines both for maximum effectiveness.

We've covered semantic search and keyword search separately, but each has critical failure modes. Hybrid search merges them to catch edge cases that either approach would miss.

The Trade-Offs Recap

Keyword Search (BM25)

Strengths:

• Exact term matching (IDs, codes, acronyms, nouns)
• High precision: results are usually relevant
• Works well for specific identifiers

Weaknesses:

• Low recall: misses synonyms, paraphrases, and concept matches
• Query: "car problems" won't match "automobile issues" or "vehicle troubles"

Semantic Search (Vector)

Strengths:

• High recall: captures conceptual matches
• Query: "how to cook pasta" matches "boiling noodles guide"
• Understands meaning through encoder models

Weaknesses:

• Low precision: can return noisy results
• Fails on exact identifiers not encoded in the model
• Query: "Error code E4021" fails if E4021 wasn't in training data
• Query: "John Smith contact" fails without explicit encoding

Precision vs Recall

When measuring retrieval systems, we use two key metrics:

Precision: Of the results I got, how many are relevant? Recall: Of all relevant documents that exist, how many did I find?

Hybrid search optimizes for both: find everything relevant (recall) while keeping results clean (precision).

When Each Approach Fails

Keyword Search Failure Modes

Keyword matching is limited by the exact terms in your index. No synonym expansion, no concept understanding.

Semantic Search Failure Modes

Semantic search doesn't understand things outside the encoder's training distribution.

Score Fusion Strategies

Hybrid search combines indexes through score fusion, merging results from multiple retrieval methods. Mixpeek supports five fusion techniques:

1. Reciprocal Rank Fusion (RRF)

The default and most common approach.

Combines based on position, not scores. Merges different indexes on query time by ranking.

Best for: General-purpose hybrid search when scores aren't comparable.

How it works:

Documents appearing high in multiple result sets get boosted.

2. Distribution-Based Score Fusion

Normalizes scores based on the distribution of results in each index.

Best for: When your indexes have varying score distributions (e.g., one index returns scores 0-1, another returns 0-100).

How it works:

• Calculate mean and standard deviation per index
• Normalize scores to z-scores
• Merge normalized scores

3. Weighted Combination

Assign explicit weights to each index.

Best for: When you know one index is more important than another for your use case.

Example:

hybrid_score = (0.7 × semantic_score) + (0.3 × keyword_score)

You can also weight multiple semantic indexes:

# Encoder model 1 + encoder model 2 + keyword
hybrid_score = (0.5 × clip_score) + (0.3 × bert_score) + (0.2 × bm25_score)

4. Maximum Score

Take the highest score across all indexes for each document.

Best for: When you want documents that excel in at least one retrieval method.

How it works:

final_score = max(semantic_score, keyword_score, other_index_score)

5. Learned Fusion

Use signals from user behavior to programmatically determine weights.

Most powerful but most complex.

Signals include:

• Clicks, bookmarks, views
• Dwell time per result
• Attribution to specific indexes

How it works:

• Track which index produced clicked results
• Use ML to learn optimal weights over time
• Adapt fusion strategy based on query patterns

Example scenario:

Hybrid Search in Practice

Example: Developer Documentation Search

Query: "OAuth2 setup problems"

Analysis:

• "OAuth2" → Exact acronym → keyword search
• "setup problems" → Conceptual → semantic search

Results:

Hybrid catches:

• Exact matches: "OAuth2" keyword match
• Semantic matches: "authentication configuration issues", "token generation failures"
• Edge cases: "PKCE flow" (acronym needing keyword) AND "secure login flow" (needing semantic)

The 80/20 Rule

Hybrid search is about catching edge cases.

• 80% of queries work fine with semantic search alone
• 20% need keyword precision (acronyms, IDs, exact terms)

But that 20% matters:

• Error codes
• Product SKUs
• Legal citations
• Medical codes (ICD-10)
• Technical acronyms (PKCE, JWT, OAuth2)

Without hybrid search, you miss critical exact-match requirements.

Building Hybrid Search in Mixpeek

Mixpeek enables hybrid search through multi-index retrieval stages.

Studio: Feature Filter + Attribute Filter

# Create hybrid retriever
retriever = client.retrievers.create(
    name="hybrid-search",
    collection="scene-segments",
    stages=[
        {
            # Semantic search (vector)
            "type": "feature_search",
            "feature_uri": "multimodal-extractor-v1",
            "input_mode": "text",
            "limit": 20
        },
        {
            # Keyword search (attribute filter)
            "type": "attribute_filter",
            "query": "OAuth2 OR PKCE",
            "operator": "text_search"
        }
    ]
)

API: Multi-Stage Retriever

# Execute hybrid search with RRF fusion
results = client.retrievers.execute(
    retriever_id="hybrid-search",
    inputs={
        "text": "OAuth2 setup problems"
    },
    fusion={
        "strategy": "reciprocal_rank_fusion",
        "k": 60
    }
)

Weighted Fusion Example

# Configure weighted combination
results = client.retrievers.execute(
    retriever_id="hybrid-search",
    inputs={
        "text": "OAuth2 setup problems"
    },
    fusion={
        "strategy": "weighted",
        "weights": {
            "feature_search": 0.7,
            "attribute_filter": 0.3
        }
    }
)

Real-World Example: Developer Docs

Scenario: Searching developer documentation for authentication issues.

Query: "OAuth2 setup problems"

Hybrid Search Pipeline:

1. Feature Search (Semantic):

   • Finds: "authentication configuration issues"
   • Finds: "token generation failures"
   • Finds: "secure login flow"
   • Returns: 89 documents

2. Attribute Filter (Keyword):

   • Finds: Exact "OAuth2" mentions
   • Finds: "PKCE flow" (specific acronym)
   • Returns: 12 documents

3. Fusion (RRF):

   • Merges both result sets
   • Boosts documents appearing in both
   • Final: 94 documents with 91% relevance

Edge Case Captured: "PKCE flow" requires keyword search (acronym) but "secure login flow" requires semantic search (concept). Hybrid catches both.

Multiple Encoder Fusion

Hybrid search isn't limited to keyword + semantic. You can fuse multiple semantic indexes:

# Combine 3 different encoder models
stages = [
    {
        "type": "feature_search",
        "feature_uri": "clip-v1",  # Multimodal encoder
        "limit": 20
    },
    {
        "type": "feature_search",
        "feature_uri": "bert-v1",  # Text-only encoder
        "limit": 20
    },
    {
        "type": "feature_search",
        "feature_uri": "custom-domain-encoder",  # Your fine-tuned model
        "limit": 20
    }
]

# Fuse with weighted combination
fusion = {
    "strategy": "weighted",
    "weights": {
        "clip-v1": 0.5,
        "bert-v1": 0.3,
        "custom-domain-encoder": 0.2
    }
}

Sparse + Dense + Dense:

• Sparse: Keyword (BM25)
• Dense: Semantic (CLIP)
• Dense: Semantic (BERT)

All can be combined in a single hybrid retriever.

Key Takeaways

1. Keyword search has high precision, low recall
2. Semantic search has low precision, high recall
3. Hybrid search combines both for high precision AND high recall
4. Reciprocal Rank Fusion (RRF) is the default fusion strategy
5. Weighted fusion lets you prioritize specific indexes
6. Learned fusion adapts weights based on user behavior signals
7. The 80/20 rule: Hybrid catches edge cases semantic alone misses
8. Mixpeek hybrid search: feature_filter (vector) + attribute_filter (keyword)

What's Next?

With hybrid search fundamentals covered, explore:

• Reranking Strategies: LLM reranking for final result ordering
• Query Expansion: Synonym handling and query augmentation
• Cross-Modal Retrieval: Searching across video, image, text, and audio

Resources

• Mixpeek Retrievers Documentation
• Feature Search Stage
• Attribute Filter Stage
• Fusion Strategies

Discussion Questions

• When would you prefer keyword-only search over hybrid search?
• How would you determine the optimal weights for weighted fusion in your domain?
• What user behavior signals would be most valuable for learned fusion?