Video Understanding: From Frames to Contextual Search

Video understanding isn't just image understanding at scale: it's capturing temporal context that single frames can never provide.

You've learned how to encode images into embeddings. But what happens when you need to search video content? A person dribbling a basketball looks like "a person running with a ball" in a single frame. Video understanding captures the motion: the dribble, the jump, the dunk.

What Makes Video Different?

A video is composed of three core components:

1. Visual Frames: Still images captured at the frame rate (30fps, 60fps, etc.)
2. Audio Tracks: Speech, music, sound effects
3. Metadata: Duration, timestamps, capture source information

The challenge: videos have redundancy. At 60fps, consecutive frames are nearly identical. We need strategies to extract meaningful content without processing every pixel.

Frame Extraction Techniques

Before we can understand video, we need to extract the frames. There are three primary approaches:

1. Uniform Sampling

Extract frames at fixed intervals: every second, every half-second, etc.

# Extract one frame per second from a 60fps video
sampling_rate = 1  # seconds
frames = video.extract_frames(interval=sampling_rate)

# 10-minute video → 600 frames

Pros:

• Simple to implement
• Predictable output size
• Fast processing

Cons:

• May miss important scenes between samples
• Doesn't understand visual changes
• Treats all intervals equally (boring scenes = action scenes)

2. Keyframe Detection

Capture frames at peak action moments or significant visual changes.

# Detect keyframes based on visual change threshold
keyframe_threshold = 0.7
keyframes = video.detect_keyframes(threshold=keyframe_threshold)

# Returns frames where visual content changed significantly

Pros:

• Captures important moments
• Reduces redundancy
• Action-oriented

Cons:

• Variable output size
• Computationally more expensive
• May miss subtle but important content

3. Scene-Based Detection

The most sophisticated approach: detect when scenes actually transition.

How it works:

1. Measure pixel differences between consecutive frames
2. When the delta exceeds a threshold, mark a scene boundary
3. Extract representative frames from each scene

# Scene detection measures pixel transitions
frame_a = video[t=0]
frame_b = video[t=1]

pixel_delta = calculate_difference(frame_a, frame_b)

if pixel_delta > scene_threshold:
    # New scene detected
    mark_scene_boundary()

Tools for scene detection:

• AutoShot: State-of-the-art shot boundary detection model
• PySceneDetect: Open-source scene detection library
• Semantic Deduplication: Compare frame embeddings: when cosine similarity drops below threshold, mark as new scene

# Semantic deduplication approach
embedding_1 = model.encode(frame_1)
embedding_2 = model.encode(frame_2)

similarity = cosine_similarity(embedding_1, embedding_2)

if similarity < 0.5:
    # Semantically different content - new scene
    create_scene_boundary()

Pros:

• Semantically meaningful segments
• Preserves complete scenes
• Optimal for content understanding

Cons:

• Variable segment lengths
• Requires ML models
• Higher computational cost

Frame-Level vs Video-Level Embeddings

Once you have frames, you have a critical architectural choice: embed each frame independently or embed the sequence together.

Frame-Level Embeddings

Treat each extracted frame as an independent image.

# Frame-level: Embed each frame separately
for frame in extracted_frames:
    embedding = siglip_model.encode(frame)
    index.add(embedding)

# Query with text
results = index.search("person with basketball")

What frame-level captures:

• Objects in the scene
• Visual composition
• Static concepts

What frame-level misses:

• Motion and action
• Temporal relationships
• Transitions between frames

Example limitation:

Video-Level Embeddings

Encode the sequence of frames together into a single embedding.

# Video-level: Embed frame sequence as one unit
frame_sequence = [frame_1, frame_2, frame_3, frame_4]
embedding = video_model.encode(frame_sequence)

# Single embedding captures the temporal context
index.add(embedding)

What video-level captures:

• Actions and motion (dribbling, jumping, landing)
• Expressions and body language
• Scene transitions
• Temporal patterns

The key insight:

Choosing Between Approaches

Trade-offs:

Video Embedding Models

Video understanding requires specialized models that process frame sequences together.

Vertex AI Multimodal Embeddings

Google Cloud's Vertex AI provides a multimodal embedding model that supports image, video, and text in the same vector space.

Key features:

• Unified embedding space for text, image, and video
• Configurable dimensions (128, 256, 512, 1408)
• Start/end offset support for video segments
• Production-ready infrastructure

# Vertex multimodal configuration
model = "multimodalembedding@001"
dimensions = 1408  # Full dimension space

# Embed video segment
embedding = vertex_model.embed(
    video=video_bytes,
    start_offset=0,
    end_offset=30  # 30-second segment
)

Dimension options:

Using Feature URIs in Mixpeek

Mixpeek abstracts the complexity of video embedding models with Feature URIs:

# Feature URI encapsulates model configuration
feature_uri = "mixpeek://multimodal-extractor/v1/vertex-multimodal-1408"

# No need to configure:
# - Vector dimensions
# - Inference endpoints
# - Model versions
# - API authentication

The Feature URI handles:

• Model selection
• Dimension configuration
• API abstraction
• Version management

Building Video Search with Scene Segmentation

Let's build a complete video understanding pipeline.

Step 1: Create a Collection with Scene Segmentation

from mixpeek import Mixpeek

client = Mixpeek(api_key="your-api-key")

# Create collection that splits videos by scene
collection = client.collections.create(
    collection_name="scene-segments",
    source_bucket="raw-videos",
    feature_extractors=[{
        "type": "multimodal",
        "split_method": "scene",  # Scene-based chunking
        "feature_uri": "mixpeek://multimodal-extractor/v1/vertex-multimodal-1408",
        "dimensions": 1408
    }]
)

When videos are uploaded:

1. AutoShot detects scene boundaries
2. Each scene is encoded using the video embedding model
3. Scene embeddings are indexed for retrieval

Step 2: Create a Retriever

# Create retriever for scene-based search
retriever = client.retrievers.create(
    retriever_name="video-semantic-search",
    collections=["scene-segments"],
    stages=[{
        "type": "feature_search",
        "feature_uri": "mixpeek://multimodal-extractor/v1/vertex-multimodal-1408",
        "limit": 20
    }]
)

Step 3: Execute Semantic Queries

# Search for contextual video content
results = retriever.execute(
    inputs={"text": "person talking in the video"}
)

# Returns scene segments with:
# - Relevance scores
# - Start/end timestamps
# - Source video reference
# - Extracted features

for result in results:
    print(f"Score: {result.score}")
    print(f"Video: {result.metadata['source_video']}")
    print(f"Timestamp: {result.metadata['start_time']} - {result.metadata['end_time']}")

Step 4: Join Across Collections

The power of decomposition: query scene segments and join with other collections:

# Scene segments can join with:
# - Time segments (fixed intervals)
# - Text embeddings (transcriptions)
# - Visual style features
# - Face detection results

results = retriever.execute(
    inputs={"text": "product demo"},
    joins=[{
        "collection": "time-segments",
        "on": "source_video_id"
    }]
)

Video Chunking: Scenes vs Time

Recall from the chunking module: video chunking is analogous to text chunking.

Time-Based Chunking

Split by fixed intervals (like splitting text by token count):

# Every 10 seconds
split_method = "time"
split_interval = 10  # seconds

Best for:

• Predictable segment sizes
• Large-scale initial processing
• Simple use cases

Scene-Based Chunking

Split by visual transitions (like splitting text by semantic similarity):

# Scene boundaries via AutoShot
split_method = "scene"
detection_threshold = 0.5

Best for:

• Semantic video retrieval
• Content with clear scene transitions
• Action and motion search

Silence-Based Chunking

Split by audio pauses (like splitting text by sentence):

# Pause detection
split_method = "silence"
silence_threshold = -40  # dB

Best for:

• Speech content (talks, lectures, podcasts)
• Idea-based segmentation
• Dialogue analysis

Image Understanding vs Video Understanding

Let's crystallize the difference:

Image Understanding

Video Understanding

What video understanding unlocks:

• Expressions: Facial movements over time
• Body language: Gestures and posture changes
• Actions: Running, jumping, dancing
• Transitions: Scene changes, camera movements
• Speech analysis: Tone and emphasis patterns

Architecture Summary

Key Takeaways

1. Video ≠ Many Images: Video understanding captures temporal context that frames alone cannot
2. Frame extraction matters: Sampling, keyframe detection, and scene-based extraction have different trade-offs
3. Video embeddings encode motion: "Dribbling" requires seeing the sequence, not just one frame
4. Scene-based chunking produces semantically meaningful segments
5. Feature URIs abstract complexity: One URI encapsulates model, dimensions, and versioning
6. Trade-offs exist: Video-level is more expensive but captures action; frame-level is cheaper but misses motion
7. Decomposition enables flexibility: Split by scene, query across collections, join on demand

What's Next?

With video understanding covered, explore:

• Audio Understanding: Speech, music, and sound as embeddings
• Multimodal Fusion: Combining video, audio, and text retrieval
• Real-Time Video Search: Streaming video analysis

Resources

• Mixpeek Video Extractor Docs
• Vertex AI Multimodal Embeddings
• AutoShot Model
• PySceneDetect
• Mixpeek Semantic Video Understanding Blog
• Reverse Video Search: How It Works + Python API Tutorial

Try It Yourself

1. Upload a video with distinct scenes to Mixpeek Studio
2. Configure scene-based chunking
3. Query with action-based text ("person walking", "product demo")
4. Compare results with time-based chunking: notice the difference in relevance

Discussion Questions

• When would you choose frame-level over video-level embeddings?
• How does scene detection compare to semantic deduplication for chunking?
• What video use cases require both audio and visual understanding?
• How would you handle very long videos (hours of content)?