The Data Transformation Pipeline

Before diving into multimodal AI, you need to understand how Mixpeek thinks about data. This isn't just architecture: it's a philosophy that shapes every decision in the platform.

The core insight: Raw files aren't searchable. Intelligence emerges through transformation.

The Fundamental Transformation

Every piece of content you work with follows the same journey:

Let's break this down:

Objects: The Raw Material

Objects are your source files: videos, images, PDFs, audio files. They're stored in Buckets, which are essentially raw storage containers.

Characteristics of Objects:

• Immutable source files
• Binary blobs (not directly searchable)
• Complete fidelity to original
• The ground truth

# Objects live in Buckets
bucket = mixpeek.buckets.create(
    name="marketing-assets",
    storage_config={
        "provider": "s3",
        "bucket": "my-company-assets"
    }
)

Documents: The Searchable Layer

Documents are what Objects become after decomposition. A single video might produce dozens of documents: one per scene, frame, transcript segment, or detected entity.

Characteristics of Documents:

• Structured, indexed data
• Live in Collections
• Searchable via vector and keyword queries
• Maintain lineage to source Objects

# Documents live in Collections
collection = mixpeek.collections.create(
    name="marketing-content",
    pipeline={
        "video": {
            "extract": ["scenes", "transcripts", "faces", "objects"]
        }
    }
)

Enriched Knowledge: The Intelligence Layer

Enriched Knowledge emerges when Documents are composed, compared, and augmented. This happens through Retrievers: composed search interfaces that span multiple Collections.

Characteristics of Enriched Knowledge:

• Multi-source insights
• Similarity-based relationships
• Aggregated intelligence
• Ready for downstream applications

# Retrievers compose knowledge
retriever = mixpeek.retrievers.create(
    name="brand-intelligence",
    sources=[
        {"collection": "marketing-content", "weight": 0.7},
        {"collection": "competitor-content", "weight": 0.3}
    ]
)

The Three-Layer Architecture

Mixpeek's architecture maps directly to this transformation:

Layer 1: Buckets (Raw Storage)

Buckets connect to your existing storage:

• S3, GCS, Azure Blob
• Local filesystems
• HTTP/HTTPS endpoints

They're passive: just pointers to where your Objects live.

Layer 2: Collections (Searchable)

Collections are active. When Objects flow in:

1. Decomposition breaks them into semantic units
2. Extraction pulls features from each unit
3. Embedding creates vector representations
4. Indexing makes everything searchable

Layer 3: Retrievers (Composed)

Retrievers are the interface layer:

• Combine multiple Collections
• Apply business logic and weighting
• Handle ranking and reranking
• Deliver unified results

Decomposition: Breaking Things Apart

The magic happens in decomposition. Complex Objects become many simple Documents.

Video Decomposition Example

A 60-second video ad might decompose into:

Original: ad_creative.mp4

• 12 scene documents (scene changes)
• 180 frame documents (3 FPS sampling)
• 24 transcript chunks (by sentence)
• 8 face documents (detected people)
• 15 object documents (products, logos)
• 1 audio embedding document

Each document is independently searchable, but all maintain lineage back to the source.

# Decomposition is configured, not coded
pipeline = {
    "video": {
        "scenes": {
            "method": "content_aware",
            "min_duration": 2.0
        },
        "frames": {
            "sample_rate": 3,  # 3 FPS
            "keyframes_only": False
        },
        "transcription": {
            "chunk_by": "sentence",
            "language": "auto"
        },
        "faces": {
            "min_confidence": 0.9,
            "recognize": True
        },
        "objects": {
            "categories": ["products", "logos", "text"]
        }
    }
}

Why Decomposition Matters

1. Granular Search: Find the exact moment, not just the video
2. Multi-Modal Alignment: Match text to visuals to audio
3. Composability: Recombine in novel ways
4. Efficiency: Only retrieve what you need

Recomposition: Putting Things Together

Decomposition creates parts. Recomposition creates intelligence.

Multi-Stage Pipelines

# Stage 1: Retrieve relevant scenes
scenes = retriever.search(
    query="product demonstration",
    filters={"type": "scene"},
    limit=50
)

# Stage 2: Enrich with face detection results
enriched = []
for scene in scenes:
    faces = collection.search(
        query={"object_id": scene.id},
        filters={"type": "face"}
    )
    scene.faces = faces
    enriched.append(scene)

# Stage 3: Aggregate by product
products = aggregate_by_product(enriched)

Retriever Composition

Retrievers can compose across Collections:

retriever = mixpeek.retrievers.create(
    name="competitive-intelligence",
    stages=[
        {
            "name": "brand_content",
            "collection": "our-ads",
            "weight": 0.5
        },
        {
            "name": "competitor_content",
            "collection": "competitor-ads",
            "weight": 0.5
        }
    ],
    merge_strategy="interleave",
    rerank={
        "model": "cross-encoder",
        "top_k": 20
    }
)

Enrichment as Immutable Joins

Here's a principle that separates Mixpeek from traditional databases:

Don't mutate. Enrich through similarity.

The Problem with Mutation

Traditional approach:

# BAD: Mutating the original record
video.brand_safety_score = 0.85
video.categories = ["automotive", "luxury"]
video.save()  # Overwrites original

Problems:

• Loses history
• Couples analysis to storage
• Can't run multiple analyses
• No provenance

The Enrichment Pattern

Mixpeek approach:

# GOOD: Enrich through similarity join
enrichment = mixpeek.enrichments.create(
    source_collection="videos",
    enrichment_type="brand_safety",
    config={
        "model": "brand-safety-v2",
        "threshold": 0.7
    }
)

# Results are a separate layer, joined by similarity
results = retriever.search(
    query="safe automotive content",
    enrichments=["brand_safety"],  # Include enrichment data
    filters={"brand_safety.score": {"$gte": 0.9}}
)

Benefits of Immutable Enrichment

1. Full History: See how classifications changed over time
2. Multiple Perspectives: Run different models on same content
3. A/B Testing: Compare enrichment strategies
4. Debugging: Trace why a result was classified a certain way

Configuration Over Code

Mixpeek pipelines are declarative. You describe what you want, not how to do it.

Declarative Pipeline Definition

# pipeline.yaml
name: video-intelligence
version: 1.0

sources:
  - bucket: marketing-assets
    patterns: ["*.mp4", "*.mov"]

decomposition:
  video:
    scenes:
      method: content_aware
      min_scene_duration: 2.0
    transcription:
      model: whisper-large
      chunk_strategy: sentence
    embeddings:
      models:
        - name: clip-vit-large
          apply_to: frames
        - name: text-embedding-3-large
          apply_to: transcripts

enrichment:
  - type: classification
    model: brand-safety-v3
    apply_to: scenes
  - type: entity_extraction
    model: product-detection
    apply_to: frames

output:
  collection: video-intelligence
  retriever: brand-content-search

Why Configuration Over Code

1. Version Control: Track pipeline changes in git
2. Reproducibility: Same config = same results
3. Auditability: See exactly what processing occurred
4. Portability: Move pipelines between environments

Lineage: Provenance as First-Class

Every Document knows where it came from. This isn't metadata: it's core to the architecture.

The Lineage Chain

Object (source video)

• Document (scene at 0:45)
  • Embedding (CLIP vector)
  • Enrichment (brand safety: 0.92)
  • Enrichment (category: automotive)

Querying Lineage

# Get the source of any document
doc = collection.get("doc_12345")
lineage = doc.lineage

print(lineage)
# {
#     "object_id": "obj_abc",
#     "object_url": "s3://bucket/video.mp4",
#     "extraction_timestamp": "2024-01-15T10:30:00Z",
#     "pipeline_version": "1.2.0",
#     "extraction_config": {...},
#     "parent_documents": ["doc_123", "doc_456"]
# }

# Navigate back to source
source = mixpeek.objects.get(lineage["object_id"])

Why Lineage Matters

1. Debugging: Why did search return this result?
2. Compliance: Prove data processing chain
3. Reproducibility: Recreate any result
4. Updates: Know what to reprocess when pipelines change

Putting It All Together

Let's trace a complete flow:

from mixpeek import Mixpeek

client = Mixpeek(api_key="your_key")

# 1. Connect raw storage (Objects)
bucket = client.buckets.create(
    name="ad-creatives",
    source="s3://my-bucket/ads/"
)

# 2. Define processing (Decomposition)
collection = client.collections.create(
    name="ad-intelligence",
    bucket_id=bucket.id,
    pipeline={
        "video": {
            "scenes": {"method": "content_aware"},
            "transcription": {"model": "whisper"},
            "embeddings": {"model": "clip-vit-large"}
        }
    }
)

# 3. Add enrichment layers
client.enrichments.create(
    collection_id=collection.id,
    type="brand_safety",
    model="brand-safety-v3"
)

# 4. Create search interface (Retriever)
retriever = client.retrievers.create(
    name="safe-ad-search",
    collection_id=collection.id,
    filters={
        "brand_safety.score": {"$gte": 0.9}
    }
)

# 5. Query enriched knowledge
results = retriever.search(
    query="energetic product demonstration",
    limit=10
)

# 6. Trace lineage
for result in results:
    print(f"Scene from: {result.lineage['object_url']}")
    print(f"Timestamp: {result.metadata['start_time']}")
    print(f"Safety: {result.enrichments['brand_safety']['score']}")

Key Takeaways

1. Objects → Documents → Knowledge: This is the fundamental transformation
2. Three Layers: Buckets (storage) → Collections (search) → Retrievers (composition)
3. Decompose to Compose: Break complex things into searchable parts, then recombine
4. Enrich, Don't Mutate: Add layers of intelligence without modifying source data
5. Configure, Don't Code: Declarative pipelines are reproducible and auditable
6. Lineage is Core: Every result traces back to its source

What's Next?

Now that you understand the mental model, you're ready to dive into the details:

• Module 1: What Is Multimodal Data?
• Module 2: From Pixels to Vectors
• Module 3: Vector Databases & Why They Matter

Resources

• Mixpeek Architecture Docs
• Pipeline Configuration Reference
• Lineage API Reference

Discussion Questions

• How does your current system handle provenance tracking?
• What would change if you stopped mutating and started enriching?
• Where in your pipeline would decomposition add the most value?