1.  [](/)
2.  [Diagrams](/diagrams)
3.  Multimodal Decomposition: One File, Many Signals

[Back to Diagrams](/diagrams)Diagram Series

# Multimodal Decomposition: One File, Many Signals

Multimodal Decomposition: One File, Many Signals

![Video highlight detection pipeline: a video is decomposed into shots and segments, each segment is scored across visual, audio, and transcript signals, contiguous high-scoring intervals merge into moments, and moments are ranked per video.](/diagrams/multimodal-decomposition.webp)Full screen

Multimodal Decomposition: One File, Many Signals

You don't search for a video. You search inside it.

Traditional search stores a video as a single record: one document, one embedding. You can find the video, but not the moment at 2:34 where someone says "quarterly revenue" next to a chart on the whiteboard.

Decomposition changes that. It breaks a file into segments, split on scene boundaries, silence gaps, or fixed time intervals, and each segment gets its own set of embeddings.

Visual: keyframes run through CLIP or SigLIP for per-frame vectors.

Audio: speech transcribed with Whisper, then embedded with E5.

Faces: detected and encoded with ArcFace, one vector each.

Text: on-screen text pulled with OCR, then embedded.

Now every segment is its own searchable document carrying several vectors. A 3 minute video can turn into 15 documents with 5 or more embeddings apiece.

The same idea carries across every content type. A PDF splits into pages and then paragraphs, each with text embeddings. An audio file splits on silence into transcript segments. An image gives you visual, OCR, and description vectors.

The payoff is that you can search by what was said, what was shown, what was written, or who showed up, at the segment level instead of the file level.

That's the real gap between multimodal search and dropping a video title into a vector database.

## Where this diagram appears

*   [Guide: video highlight detection →](/guides/video-highlight-detection)

Run this on your own data

Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API.

[Search your own data](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=diagram_detail&utm_campaign=mvs_onramp)

## Related diagrams

[

![Object decomposition: you search inside the file, not for it. A one-hour recording.mp4 is split into scenes (visual embeddings), speech (transcript embeddings on silence and speaker turns), faces (who appeared and when), and on-screen text (OCR of slides and captions). A whole file being findable is not the same as usable: the thing you need is a moment inside it, like 23,41 where the renewal number is spoken while the pricing slide is on screen. The same move works on video, PDFs, and audio: one file becomes dozens of small, independently searchable documents.](/diagrams/foundations/decomposition.webp)

You search inside the file, not for it



](/diagrams/decomposition)[

![Feature extraction turns raw pixels into searchable signals: one video frame becomes a structured row, face, scene, spoken words (transcript), on-screen text (OCR), and a visual embedding. Every question you later ask the video is answered from these extracted signals, not from the raw pixels; if a signal was never extracted, nothing downstream can search it.](/diagrams/foundations/feature-extraction.webp)

Pixels in, signals out



](/diagrams/feature-extraction)[

![Scene segmentation turns a continuous video into searchable segments in two levels. First, shot boundary detection compares consecutive frames: a hard cut spikes a histogram difference (over 1000 fps on CPU), while dissolves and fades spread the change across many frames, so adaptive detectors flag a boundary when the score beats a rolling local mean by 2 to 3 sigma. Second, shots group into scenes, where a shot is one continuous camera take and a scene is a semantic unit that can span many shots. Cleanup rules merge segments under 2 seconds into their nearest neighbor and split segments over 60 seconds at audio silence. The scene becomes the retrieval unit: one embedding per scene instead of per frame, results that carry start and end timestamps, and storage down 10 to 100x.](/diagrams/scene-segmentation.webp)

Scene Segmentation: How a Video Becomes Searchable Scenes



](/diagrams/scene-segmentation)