What Are Embeddings?

Similar meaning = similar numbers. That's the entire idea.

Embeddings are the foundation of everything in modern search. Before we can find similar content, compare meanings, or build intelligent retrieval systems, we need a way to represent data as numbers that machines can process.

The Core Insight

Text, images, and audio are fundamentally different data types. But embeddings give us a universal language:

Everything becomes a list of numbers. Similar things get similar numbers.

# Two phrases with identical meaning, zero keyword overlap
phrase_1 = "I want a refund"
phrase_2 = "Give me my money back"

# After embedding, they're nearly identical vectors
embedding_1 = model.encode(phrase_1)  # [0.23, -0.45, 0.78, ...]
embedding_2 = model.encode(phrase_2)  # [0.22, -0.44, 0.79, ...]

cosine_similarity(embedding_1, embedding_2)  # 0.94 - almost identical!

This is the magic: semantic similarity becomes mathematical distance.

What Embeddings Actually Are

An embedding is a fixed-length vector (list of numbers) that represents the meaning of some input.

Key properties:

• Fixed length: Every input becomes the same size vector
• Dense: Most values are non-zero (for dense embeddings)
• Learned: Neural networks learn what numbers to assign
• Semantic: Similar meanings → similar vectors

Why Embeddings Matter

Traditional search relies on keyword matching:

Embedding-based search understands meaning:

This enables:

• Semantic search: Find by meaning, not just keywords
• Cross-modal retrieval: Search images with text queries
• Clustering: Group similar content automatically
• Recommendations: Find "more like this"

Dense vs Sparse Embeddings

Not all embeddings are created equal. Mixpeek supports both types:

Dense Embeddings (ColBERT)

Every dimension has a meaningful value. The entire vector participates in similarity.

# Dense embedding example
dense_vector = [0.12, -0.34, 0.56, 0.78, -0.23, 0.45, ...]
# Length: 768 dimensions
# Non-zero values: ~768 (all of them)

Characteristics:

• Compact representation
• Great for semantic similarity
• Works across modalities
• Computationally efficient for retrieval

Best for:

• Semantic search
• Cross-modal matching
• Finding conceptually similar content

Sparse Embeddings (SPLADE)

Most dimensions are zero. Only specific "activated" dimensions have values.

# Sparse embedding example
sparse_vector = {
    142: 0.85,   # "refund" concept
    3847: 0.72,  # "money" concept
    9821: 0.63,  # "request" concept
    # ... most dimensions are 0
}
# Length: 30,000+ dimensions
# Non-zero values: ~100-200

Characteristics:

• Interpretable (dimensions map to concepts)
• Excellent for exact term matching
• Handles rare terms well
• Sparse storage efficient

Best for:

• Keyword-sensitive search
• Domain-specific terminology
• When exact terms matter

Combining Both in Mixpeek

# Configure both embedding types for a collection
collection = mixpeek.collections.create(
    name="support-tickets",
    feature_extractors=[
        {
            "name": "dense_embeddings",
            "type": "colbert",
            "model": "colbert-ir/colbertv2.0"
        },
        {
            "name": "sparse_embeddings",
            "type": "splade",
            "model": "naver/splade-cocondenser"
        }
    ]
)

# Search uses both for best results
results = retriever.search(
    query="refund request for damaged item",
    embedding_types=["dense", "sparse"],
    weights={"dense": 0.7, "sparse": 0.3}
)

What Do 768 Dimensions Represent?

A common question: "What does dimension 147 mean?"

The honest answer: We don't know exactly. And that's okay.

The Intuition

Think of dimensions as learned "concepts" the model discovered during training:

• Some dimensions might capture sentiment (positive/negative)
• Some might capture topic (technical, casual, formal)
• Some might capture entities (people, places, things)
• Most capture abstract patterns we can't name

A Useful Mental Model

Imagine a 3D space where:

• X-axis = formality (casual ↔ formal)
• Y-axis = sentiment (negative ↔ positive)
• Z-axis = topic (technical ↔ creative)

"Happy customer feedback" might be at coordinates (0.3, 0.8, -0.2):

• Somewhat casual
• Very positive
• Slightly non-technical

Now scale this to 768 dimensions. Each dimension captures some aspect of meaning, and together they form a rich semantic fingerprint.

Visualizing High-Dimensional Space

We can project 768 dimensions down to 2D or 3D for visualization:

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

# Get embeddings for support tickets
embeddings = [model.encode(ticket) for ticket in tickets]

# Project to 2D
tsne = TSNE(n_components=2, random_state=42)
projected = tsne.fit_transform(embeddings)

# Plot - similar tickets cluster together!
plt.scatter(projected[:, 0], projected[:, 1])
plt.title("Support Ticket Clusters")

When you do this with real data, you'll see:

• Billing issues cluster together
• Technical problems form another cluster
• Feature requests group separately

The model learned these categories without being told they exist!

The Support Ticket Demo

Let's make this concrete with a real example:

from mixpeek import Mixpeek

client = Mixpeek(api_key="your_key")

# Create collection with embeddings
collection = client.collections.create(
    name="support-tickets",
    feature_extractors=[
        {"type": "colbert", "apply_to": "text"}
    ]
)

# Index 10,000 support tickets
for ticket in tickets:
    collection.insert({
        "text": ticket.content,
        "metadata": {"category": ticket.category}
    })

# Now search semantically
results = collection.search(
    query="refund request",
    limit=5
)

# Results include tickets like:
# - "I want my money back" (similarity: 0.94)
# - "Please process a return" (similarity: 0.89)
# - "How do I get reimbursed?" (similarity: 0.87)
# None of these contain "refund" or "request"!

The aha moment: Similar meaning produces similar numbers, regardless of the actual words used.

Embedding Models in Mixpeek

Mixpeek supports multiple embedding models for different use cases:

Configuring Embeddings

# Text-focused collection
text_collection = mixpeek.collections.create(
    name="documents",
    feature_extractors=[
        {"type": "colbert", "apply_to": "text"},
        {"type": "splade", "apply_to": "text"}
    ]
)

# Multimodal collection
multimodal_collection = mixpeek.collections.create(
    name="media",
    feature_extractors=[
        {"type": "clip", "apply_to": "image"},
        {"type": "colbert", "apply_to": "text"},
        {"type": "whisper", "apply_to": "audio"}
    ]
)

Measuring Similarity

Once we have embeddings, how do we compare them?

Cosine Similarity

The most common metric. Measures the angle between vectors (ignoring magnitude).

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Range: -1 to 1
# 1.0 = identical meaning
# 0.0 = unrelated
# -1.0 = opposite meaning (rare in practice)

Dot Product

Faster to compute. Used when vectors are already normalized.

def dot_product(a, b):
    return np.dot(a, b)

Euclidean Distance

Measures straight-line distance. Smaller = more similar.

def euclidean_distance(a, b):
    return np.linalg.norm(a - b)

Mixpeek handles this automatically: you just specify the metric:

results = collection.search(
    query="refund request",
    metric="cosine",  # or "dot", "euclidean"
    limit=10
)

Key Takeaways

1. Embeddings = numbers that represent meaning: Fixed-length vectors capture semantic content
2. Similar meaning → similar vectors: This is the foundation of semantic search
3. Dense vs sparse: ColBERT for semantics, SPLADE for keywords: use both
4. 768 dimensions: Abstract learned concepts that together capture meaning
5. Cosine similarity: The standard way to compare embeddings

What's Next?

Now that you understand embeddings, you're ready to learn:

• Module 2: Vector Databases & Why They Matter
• Module 3: From Pixels to Vectors (visual embeddings)
• Module 4: Retrieval-Augmented Generation (RAG)

Resources

• Mixpeek Embedding Models
• ColBERT Paper
• SPLADE Paper
• Understanding Cosine Similarity

Try It Yourself

1. Take 100 customer emails and embed them: do similar issues cluster together?
2. Compare dense vs sparse results for "urgent billing problem"
3. Visualize your embeddings in 2D: what clusters emerge?

Discussion Questions

• When would you choose sparse over dense embeddings?
• How might embedding quality affect your search results?
• What happens when you search across languages with the same model?