Video Scene Search
Find specific scenes within videos using natural language descriptions. The pipeline detects scene boundaries, generates embeddings for each scene, and enables precise timestamp-level search across an entire video library. Query for visual content, actions, or spoken dialogue.
How do you search a video for a specific scene?
Split each video into scenes, embed every scene, and search those embeddings with a text description. A scene detector marks where the picture changes. Each scene gets an embedding of its frames and one of its transcript, and a query like "person presenting a chart on a whiteboard" is matched against both. Each result is the video ID with the scene's start and end time, and a reranker orders the top scenes by what was said in them.
Updated 2026-10-10
| Segmentation | Where it cuts | Works well for | Watch out for |
|---|---|---|---|
| Scene changes (split_method: scene) | Where FFmpeg's frame-difference score passes a threshold (0.5 by default; lower finds more scenes) | Edited footage: film, TV, ads, sports | Slow fades and dissolves can be missed; static footage gives very long scenes |
| Fixed windows (split_method: time) | Every N seconds | Lectures, surveillance, footage with few cuts | A moment that straddles two windows is split in half |
| Silence (split_method: silence) | At pauses where audio drops below a dB level (-40 by default) | Talks, interviews, podcasts on video | Music or constant background noise leaves few places to cut |
| Frame sampling | One frame every second or so, each embedded on its own | Finding a moment that looks like a single still image | Loses motion and dialogue; far more vectors to store |
from mixpeek import Mixpeekclient = Mixpeek(api_key="YOUR_API_KEY", namespace="video-scenes")# 1. A bucket for the videos, and a collection that detects scene boundaries and embeds each scenebucket = client.buckets.create(bucket_name="videos",bucket_schema={"properties": {"video": {"type": "video",},},},)collection = client.collections.create(collection_name="video_scenes",source={"type": "bucket", "bucket_ids": [bucket["bucket_id"]]},feature_extractor={"feature_extractor_name": "multimodal_extractor","version": "v1","parameters": {"split_method": "scene","scene_detection_threshold": 0.5,"run_transcription": True,"run_transcription_embedding": True,},},)# 2. Upload and processclient.buckets.upload(bucket["bucket_id"],blobs=[{"property": "video", "type": "video", "data": "s3://your-bucket/videos/training-session.mp4"}],)client.collections.trigger(collection["collection_id"])# 3. Scene search over visual and transcript embeddings, then a rerank against each scene transcriptretriever = client.retrievers.create(retriever_name="scene-search",collection_identifiers=["video_scenes"],input_schema={"query": {"type": "text","required": True,},},stages=[{"stage_name": "search","stage_id": "feature_search","parameters": {"searches": [{"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding","query": {"input_mode": "text","value": "{{INPUT.query}}",},"top_k": 50,},{"feature_uri": "mixpeek://multimodal_extractor@v1/multilingual_e5_large_instruct_v1","query": {"input_mode": "text","value": "{{INPUT.query}}",},"top_k": 50,},],"fusion": "rrf","final_top_k": 50,},},{"stage_name": "rerank","stage_id": "rerank","parameters": {"inference_name": "BAAI__bge_reranker_v2_m3","query": "{{INPUT.query}}","document_field": "transcription","top_k": 10,},},],)# 4. Searchresults = client.retrievers.execute(retriever["retriever_id"],inputs={"query": "person presenting a chart on a whiteboard",},)for doc in results["documents"]:print(doc["source_object_id"], doc["start_time"], doc["end_time"], doc["score"])
Feature Extractors
Multimodal Extractor
Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.
Retriever Stages
feature search
Search and filter documents by vector similarity using feature embeddings
rerank
Rerank documents using cross-encoder models for accurate relevance
Frequently asked questions
How are scene boundaries detected?
The multimodal extractor runs FFmpeg's scene-change filter, which scores how different each frame is from the one before it and starts a new scene where the score passes scene_detection_threshold (0.5 when you leave it unset). A lower threshold cuts more often and a higher one cuts less. Hard cuts are caught reliably. Slow fades and dissolves change the picture a little at a time and can stay under the threshold.
Why does my video search return the wrong scene?
Most often the segments are the wrong size. A segment that spans two moments gets one embedding that blends both, so it matches either query weakly, and a segment that is too short loses the context that identifies it. Look at the scene lengths you actually get, move the threshold, and search the transcript embedding beside the visual one. The video scene segmentation guide covers the trade-offs in depth.
Can I search both what is shown and what is said?
Yes. With run_transcription and run_transcription_embedding on, each scene document carries a visual embedding and a transcript embedding. The retriever in the code above searches both, fuses the two result lists with reciprocal rank fusion, and reranks the top 50 against each scene transcript.
What does a search result look like?
Each result is one scene: the ID of the video it came from, its start_time and end_time in seconds, and a score. A player can seek straight to the moment instead of opening the whole file.
How long is a typical scene?
It depends on the footage. Edited film, TV and ads often cut every few seconds, while a presentation or a fixed surveillance camera can hold one shot for minutes. Footage with few cuts is usually better split by time or by silence.
Related Recipes & Resources
Explore these related resources to deepen your understanding and discover more powerful features
Multimodal Extractor
Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.
Hierarchical Classification
Assign content to multi-level category hierarchies using embedding-based classification. Define your taxonomy once, then classify new content automatically with confidence scores.
Brand Safety & Ad Verification Pipeline
Brand safety pipeline for ad networks. Scores video and image creatives against the GARM brand safety floor and suitability categories, which vendors still use after GARM wound down in 2024, before an ad is served.
Automated Video Tagging
Automatically generate descriptive tags for video content using scene analysis, object detection, and taxonomy classification. Each video receives structured labels for scenes, objects, actions, and custom business categories without manual annotation.
Searchable Video Library
Turn an unstructured video archive into a fully searchable library. Each video is decomposed into scenes with transcriptions, visual embeddings, and metadata. Users search by natural language and jump directly to the relevant moment in any video.
Semantic Multimodal Search
Unified semantic search across all content types. Query by natural language and retrieve relevant video clips, images, audio segments, and documents based on meaning-not keywords or manual tags.